HingeConstraint refuses to attach properly

I have 2 parts which are held together by a HingeConstraint; I want to make Part1 (the ball) the one that rotates, I have tried to achieve that for over an hour to no avail. The constraint always seems to prioritize Part2 (the red block inside).

The issue is not caused by the BodyGyro

Any help would be appreciated