Convert ObjectSpace Rotation Back To World Space

I have a script that clamps the cameras movement, I want to make its origin rotate with the Humanoid Root Part but it only works for one axis.

Code:

local RunService = game:GetService("RunService")
local camera = workspace.CurrentCamera
local hrp = game.Players.LocalPlayer.Character:WaitForChild("HumanoidRootPart")
RunService:BindToRenderStep("UpdateLoop", Enum.RenderPriority.Camera.Value, function()
  local rX, rY, rZ = camera.CFrame:ToOrientation()
  local limX = math.clamp(math.deg(rX), -45, 45)
  local limY = math.clamp(math.deg(rY), -45, 45)
  camera.CFrame = CFrame.new(camera.CFrame.p) * CFrame.fromOrientation(math.rad(limX), math.rad(limY), rZ)
end)

Ive tried converting it to object space, clamping, then back to world space. But the camera just breaks.
Code:

local RunService = game:GetService("RunService")
local camera = workspace.CurrentCamera
local hrp = game.Players.LocalPlayer.Character:WaitForChild("HumanoidRootPart")
RunService:BindToRenderStep("UpdateLoop", Enum.RenderPriority.Camera.Value, function()
  local relativeCF = camera.CFrame:ToObjectSpace(hrp.CFrame)
  local rX, rY, rZ = relativeCF:ToOrientation()
  local limX = math.clamp(math.deg(rX), -45, 45)
  local limY = math.clamp(math.deg(rY), -45, 45)
  local limCF = CFrame.new(relativeCF.Position) * CFrame.fromOrientation(math.rad(limX), math.rad(limY), rZ)
  local worldCF = limCF:ToWorldSpace()
  local wX, wY, wZ = worldCF:ToOrientation()
  local newCF = CFrame.new(camera.CFrame.Position) * CFrame.fromOrientation(wX, wY, rZ) 
  camera.CFrame = newCF
end)