Orient right/forward CFrame vectors correctly

Hi, I am trying to make some sort of gravity thing, where the character’s gravity changes based on the ground’s normal. I also calculate the right and forward vector perpendicular to up vector (i.e. the ground normal), so the player can move right and forward properly.

The issue is that I’m guessing when the surface’s up normal changes a quadrant or something, the right/forward vectors flip. So the camera will rotate around the player without the user doing anything, since the orientation of the camera depends on those vectors. In the video, this happens between 0:09 and 0:14

Here is the code responsible for calculating the correct right/forward vectors: (gravityDir is the down vector)

local function setGravDir(newDir)
	gravityDir = newDir
	local upVec = -newDir
	--movementFwd = Vector3.new(0, 0, upVec.Y >= 0 and 1 or -1)
	movementFwd = Vector3.new(0, 0, movementFwd.Z >= 0 and 1 or -1)
	local cross = upVec:Cross(movementFwd)
	if cross.Magnitude > 0 then
		--movementRt = (upVec.X >= 0 or upVec.Y < 0) and cross.Unit or -cross.Unit
		--movementRt = movementRt.X >= 0 and cross.Unit or -cross.Unit
		movementRt = cross.Unit
		--movementRt = (upVec.X >= 0 or upVec.Y < 0) and Vector3.new(0, -movementFwd.Z, 0) or Vector3.new(movementFwd.Z, 0, 0)
		movementRt = Vector3.new(1, 0, 0)
	movementFwd = movementRt:Cross(upVec).Unit

I tried flipping certain numbers based on the sign of certain vector components, playing the game to see if it worked, and if it didn’t, change it a bit. By doing this, I sometimes got it to work on certain surfaces. But not in all of the surfaces I tested.

Also in the code I am able to access the previous right/forward/up vectors, but I can’t figure out how to make them affect the new result so that the orientation is fixed.