How to deal with ping when making a bumper ball minigame

Hi devs!, so ive tried making a game based on a mario party minigame where you push others off a tower using boulders heres an example: Mario Party Bumper Balls , ive done the ball movement but after testing i found a problem:

I know this has to do with client replication but how would i fix this delay?

Thanks for reading.

I think best solution is to run all ball physics on server.

Yes but if I do that also the movement will have delay

Yes, but if you make the player adjust a force, and not a velocity, then it gradually increase speed and the effect is not that visible.