The more digits you have to the left of the decimal point the less you can accurately have to the right of it. It makes sense that once you have enough data to the left you end up losing rounding precision on the right which causes jumpiness.
Roblox Vector3s use single precision floating point to my knowledge, so it’s not able to represent as much as you’d think. About 7 decimals is about where things get wacky.
At least this shows that roblox has made notice of the issue, but it’d still be nice to see some official documentation making developers aware of such limitations and their exact nature.
As many others have said here, this is something to do with Floating Point errors
Floating Point can be super annoying when creating a game with a large map or something like that but I found a game that managed to get around this problem in an interesting way.
The game is called AirX 2020, it’s made by Widgeon and a few other developers and what he did to overcome this problem was to implement a recentering system.
The Fix
I’m not exactly sure how it works but I’m pretty sure the player goes to 0, 0, 0 and the place the player was at before being teleported would get teleported next to the player.
Maybe you could use something like that to help with your problem.
This is often called a floating or shifting origin. In reality it’s not the origin that’s moving but everything else. Its a solution used by many open world single player games. It works great for single player or cooperative multi-player if you can limit the distance players can stray from the group.
For mmo games you can still use this technique to handle larger map sizes but it’s common to break up the world into separate maps and use cut scenes or come up with some reasonable explanation for transitioning between maps.
Unfortunately origin shifting isn’t something Roblox offers out of the box but it does provide the tools to build it yourself. It’s not trivial though.