ToObjectSpace changing position after aim

My ToObjectSpace changes the position after i move my camera and aim again.

Code:

aimcf = aimcf:Lerp(cam.CFrame:ToObjectSpace(Gun.AimPart.CFrame):Inverse(),1)

this is because when you sway you arent at center, and then you sway back to center so where you previous aimed its not at center. I have this problem with my gun system too like when a fire then aim quick then it glitches the aim position and messes it upp.Im not sure how to solve this sadly.