Physics based puzzle breaking if too many players are using it

I have it so that the minecarts move if the player is holding and dragging them with alignposition and a railway that acts like a prismatic constraint with hinges. However, if multiple players are quickly switching between network ownerships it causes the puzzle to break and explode. Is there a way to fix this without making them be on the server?