Limit camera movement relative to where a part is facing

Help and Feedback Scripting Support
1

My problem is that when I use something like this:

	local x,y,z = cameraPart.CFrame:ToOrientation()

	local clampedX = math.clamp(math.deg(x), -30,30)
	clampedX = math.rad(clampedX)

	local clampedY = math.clamp(math.deg(y), -90, 90)
	clampedY = math.rad(clampedY)

	cameraPart.CFrame = CFrame.new(cameraPart.CFrame.Position) * CFrame.fromOrientation(clampedX,clampedY,0)

It limits camera rotation relative to the world.

I have a system in place, which moves me around the map when I press A/D:
ad

When I reach the corner of the map, it only allows my camera to move 90 degrees from where the part was facing originally:
rotate

I need to make it limit the horizontal movement by 90 degrees in both directions relative to where the part is facing, how would I go about doing this?

(sorry for the laggy gifs, hopefully you are still able to understand my issue)

Edit: the part that my camera is attached to, always faces the red square in the middle