Wall climbing forces player upwards always

Hey so I have this issue where when I climb on a wall with my wall climbing script it always slightly forces the player up. Luckily the upward force is small enough to where my downward force beats it however when I stop moving during the climb it unfortunately does this annoying rubberband effect where it slightly pushes you up. How can I fix this? I managed to fix it when going sideways by just having a YLevel variable that can only change while moving up or down but when going down I have no idea how to fix this. All help is greatly appreciated!

Here is an example: (as you can see I slightly move up when I stop moving down)
https://gyazo.com/6aeeea36c58a79c0d07b0a9af58db0aa

Here is the code:

local alignPos = Instance.new("AlignPosition")
				local climbPos = rc.Position - (rc.Position - char.HumanoidRootPart.Position).Unit*1.5
				
				-- setup alignpos
				alignPos.MaxVelocity = 100000
				alignPos.Responsiveness = 10
				alignPos.Mode = Enum.PositionAlignmentMode.OneAttachment
				alignPos.Attachment0 = char.HumanoidRootPart:WaitForChild("RootRigAttachment")
				alignPos.Parent = char.HumanoidRootPart
				alignPos.Position = climbPos
				
				-- variables
				
				local setMoving = false
				local climbSpeed = baseClimbSpeed 
				local YLevel = char.HumanoidRootPart.Position.Y
				
				-- functions
				local function setClimbDirection(direction)
					local direction = direction*climbSpeed
					climbPos = (Vector3.new(rc.Position.X,YLevel,rc.Position.Z) + direction) - (rc.Position - char.HumanoidRootPart.Position).Unit*1.5
				end
				
				-- climb loop
				repeat
					setSpeed(0)
					humanoid.AutoRotate = false
					
					char.HumanoidRootPart.CFrame = CFrame.lookAt(char.HumanoidRootPart.Position,Vector3.new(char.HumanoidRootPart.Position.X - rc.Normal.X,char.HumanoidRootPart.Position.Y,char.HumanoidRootPart.Position.Z - rc.Normal.Z))
					
					if setMoving == true and climbUp == false and climbDown == false and climbLeft == false and climbRight == false then
						climbPos = Vector3.new(rc.Position.X,YLevel,rc.Position.Z) - (rc.Position - char.HumanoidRootPart.Position).Unit*1.5
						setMoving = false
					elseif climbUp == true or climbDown == true or climbLeft == true or climbRight == true then
						
						local direction = Vector3.new(0,0,0)
						
						if climbUp == true then
							YLevel = char.HumanoidRootPart.Position.Y
							direction += Vector3.new(0,1,0)
						elseif climbDown == true then
							YLevel = char.HumanoidRootPart.Position.Y
							direction -= Vector3.new(0,1,0)
						end
						
						if climbLeft == true then
							direction -= char.HumanoidRootPart.CFrame.RightVector
						elseif climbRight == true then
							direction += char.HumanoidRootPart.CFrame.RightVector
						end
						
						setMoving = true
						
						setClimbDirection(direction)
						
					end
					
					alignPos.Position = climbPos
					
					runService.Heartbeat:Wait()
					rc = raycastIgnorePlayers(char,char.HumanoidRootPart.Position,char.HumanoidRootPart.CFrame.lookVector*2.5)
				until climbing == false or stuntime.Value > 0 or rc == nil

I know I can anchor the humanoidrootpart while the character doesnt need to be moving to temporarily solve this but I was told its not the best practice to anchor the characters humanoidrootpart which is why I dont want to do that exactly. And there would still be an upward force applied while going down which makes going down a bit slower.