From my experience, chunk loading can slow things down (ik, sad right LOL) so make sure to always double check the speed difference and try using streaming enabled
(Or have everything on the client side and have the server just tell the client manually when a change is made if that works for your game)
btw in my use cases all the parts were anchored so it might be different for you
That’s what I am doing. It’s all run on the client. None of the map will actually change except for vehicles and players, which are being accounted for with scripting.