Well with my progress of making a custom camera system, stumbled upon this. Camera is all fine, its the billboard guis that freak out with it. Without it, no problems.
But why though is a question i need to figure out.
To see scripts and overall reproduce the issue, here is the rbxl of a baseplate with one billboard gui and the camera system.
[deleted :p]
Led me to something new, billboard guis happen to be affected by mouse movement. Even though the camera rotation is 0,0,0 at all times!
Gonna continue digging into it, if someone finds anything to fix this, please do.