Custom Control and Collisions act glitchy

I made my own Character Controller that works by moving the Character’s CFrame, but of course, I also need to limit these movements to prevent the character from walking through walls. I tried that by Raycasting in all 4 main directions and also the direction of movement, which works relatively fine, however, it has a glitchy look since when the ray hits, it teleports the character back slightly, then lets it move again, then teleports it back again etc.

The script:

-- In RenderStepped(dt)

local dir = Vector3.new((math.sin(mXA)*pos.X + math.cos(-mXA)*pos.Y), 0, (math.cos(mXA)*pos.X + math.sin(-mXA)*pos.Y));
-- pos is the direction in which the player wants to move
-- mXA is the Y rotation of the camera

local objectsHit = {};
local dirRay1 = workspace:Raycast(char.LowerCollider.Position, dir * 0.5);
local dirRay41 = workspace:Raycast(char.LowerCollider.Position, Vector3.new(0.5, 0, 0));
local dirRay42 = workspace:Raycast(char.LowerCollider.Position, Vector3.new(-0.5, 0, 0));
local dirRay43 = workspace:Raycast(char.LowerCollider.Position, Vector3.new(0, 0, 0.5));
local dirRay44 = workspace:Raycast(char.LowerCollider.Position, Vector3.new(0, 0, -0.5));
-- The character is 1 stud wide, that's why I use a 0.5 long ray
	
if dirRay1 and dirRay1.Instance.CanCollide and not(table.find(objectsHit, dirRay1.Instance)) then
	table.insert(objectsHit, dirRay1.Instance);
	dir += dirRay1.Normal;
end
-- This is the same for the rest of the rays

char.LowerCollider.CFrame *= CFrame.new(dir * dt * SPEED);

I thought of casting another ray that is slightly longer than dirRay1, and if it hits but dirRay1 doesn’t, then move the character as much as there is from the wall to the end of dirRay1, but I couldn’t figure out how to achieve this.

Is there a way I can make the collisions smoother? Any help is appreciated!