Please help with my deviating ball
The parts are most likely affected by other forces, or cannot react quickly enough. Because you are using a LinearVelocity, the part has to slow down so that it is not moving the wrong direction, which takes some time.