Repro:
- 20 thousand people playing, no DataStore issues
- Shut down to fix very critical bug
- As everyone’s joining again, DataStores get throttled hard, causing an absolute mess
1 Like
You should include some information on how you use datastores. How many values per player do you save when a server shuts down? How many values per player do you load when a player joins?
What sort of issues did this cause? Is anyone reporting data loss.
Seems like loading data requires GetAsync-ing a lot of keys, which seems weird.
(you’re not one of those people that store each value in a different key, right?)
Solution: Don’t shutdown all at once, have a datastore key or something that gets changes, then when you update, just move players out of the old servers slowly, and over time it will all be updated.
A feature that would be great is a function that can lock servers from new players joining. like game:SetLocked(true)
2 Likes
slowly
Nope it’s a critical bug and needs to be fixed exactly when I want it to be fixed. Plus that’s not reliable at all. Also ROBLOX let us know we don’t need to lock/shutdown/etc old servers.
I’m not a person that GetAsyncs every key but I have a game that’s complex under the hood and needs to make more than one call per year
Have two copies of the game in a Universe. Update the two games at the same time and then gradually teleport players over to the backup server. Players are kept in-game and new players will join new servers only, while old players will be in new servers that will slowly die out.
Similar thread about datastores and weird load balancing Max Concurrent Players configuration setting
The problem though, Roblox uses amazon, and amazon has rate limits, so when you restart a massive amount of servers at all once, thats a ton of requests all going to the database, probably even overloading. When the servers all start to come back on, same thing, its a massive amount of requests in a short period of time.
So how would ROBLOX go about hypothetically solving that? No big game can ever have an emergency update?
Simple, phase out the servers over a period of time. Dont just shut down 600 servers all at once, shut them down over a period of like 10 - 30 minutes. But this is easier said than done.
If you introduce an update that breaks the game add a way to switch it off in a live server using a datastore (basically flags). Schedule your updates so that you can progressively close old servers. For example, have a specific day to update on, committing the change and then starting the phase out. If the new updates introduces an issue flick a switch in the datastore so you don’t need to shutdown any servers and repair the bug the next scheduled update.