Rotating the Character makes my VR default movents change

I am currently exploring the VR system in Roblox using a oculus 3 and my goal is to have the player’s character work for all devices including VR, to achieve this I used IKControls to make the player’s movements show when using VR inputs, the thing is that I got it working for all devices in a first person view, I even got the hand gestures showing, but for some reason when ever I rotate the HumanoidRootPart using the cameras orientation it makes the movement controls act diferent.

Is like it inverts the controllers and instead of walking forward when I rotate 90 degrees it moves left or something like that, but I am moving the joystick in the same direction, I am guessing it has to do with something not rotating properly because it acts like all the parts in the Character rotated but something is still facing the starting direction, I am guessing it could or might be some property kind of like what the Camera.HeadLocked property but am not compleatly sure.

This is the code that I use to make the player rotate to the camera direction, the weird part is that it works just fine for PC, Gamepad and even Mobile, but I have no idea why it behaves different using VR

--Services

local function rotatePlayer()
	if Humanoid.Sit or not HumanoidRootPart or not Humanoid then
		return
	end
	local direction
	local camLookVec = Camera.CFrame.LookVector
	local lookVecX, lookVecZ = camLookVec.X, camLookVec.Z
	if lookVecX ~= 0 or lookVecZ ~= 0 then
		direction = Vector3.new(lookVecX, 0, lookVecZ).Unit
	end
	Humanoid.AutoRotate = false
	local NewCFrame = CFrame.new(HumanoidRootPart.Position, HumanoidRootPart.Position + direction)
	HumanoidRootPart.CFrame = NewCFrame
end