Relative to Camera angle -Orientation?

Bare with me whilst I try to explain this:
I’m working on a project that creates a pseudo version of the ‘Galaxy’ the Player is exploring:

When the Player is in the “Pilot” seat; the world is set to be relative to the Exterior & the Camera CFrame - However, when the Player is not on the pilot seat then the world is relative to the inteior (Allowing me to add viewports and to create the immersion that the Player is infact moving throughout space.)

However, due to the way my Ship(s) work - When the Player leaves the pilot seat & NetworkOwnership is returned to the server, the ship will automatically balance itself out. (It does this by design) and because of this the Viewport adornee will also become relative to the Players now straight camera causing:

Wondering if anyone could suggest any work arounds?

CameraPart.CFrame = CF + CF.LookVector * 800
CameraPart.Size = 2*math.tan(FieldOfView/2) * 800 **ViewportSize.X/ViewportSize.Y,1,1)
Camera3.CFrame = Camera.CFrame

if Camera.CameraSubject ~= Humanoid then
  Camera2.CFrame = Exterior:GetPrimaryPartCFrame() * (Exterior:GetPrimaryPartCFrame():Inverse() * CF)