I’m currently trying to develop a game where each player can build their own house and store it forever on a tile on the map(in a small area no other player has taken yet). Basically, the issue I’m facing is that I have to save data as “chunks” instead of “houses” so it can be fetched much faster using Roblox datastores but each player can modify their house and this change needs to be saved to the chunk datastore, and be streamed to all the other active servers.
So somehow I need to create a system that lets a user modify their house in a random server out of the active ones, replicate the changes to all the other servers and update the chunk the house belongs to and save it on the datastore.
Because of Roblox’s multi-server nature, a system such as this makes me think it will be a recipe for errors and data corruption, but I also don’t have another way to achieve what I’m describing. For example, an error will happen if 2 servers try to create/update a house at almost the same time by fetching and writing to the chunk datastore causing one house change to be fully ignored(imagine for example a server fetches data before another server updates it, so if this server updates the data with a change it will ignore the other server changes unless it’s done incredibly fast and the servers aren’t a lot).
So what I’m asking is if this is a known issue in computer science or Roblox specifically, if so what are the known errors, how can I approach the problem, and if anybody tried to solve this beforehand.
if you assigned each player there own chunk, you could use profile service to load and save that chunk and not worry about each server overwritting each others data.
you can then use messaging service to have the server that the player is in tell all the other servers things such as ‘hey, a player without a chunk joined a game, and i assigned them a position at x,y,z’ or ‘hey, this player put a ancient statue right here!’ and then all the other servers wouldnt save that data, but replicate that to their server.
im not exactly a professional in this terrority of scripting, but i do hope that explanation helped you.
The problem is more complicated than that, you see a chunk is a collection of houses of multiple people(so multiple people can belong in the same chunk). For example, a chunk could contain a maximum 40x40=1600 houses that belong to different users. I already plan to have 2 data stores, a user one(that can use ProfileService) that has all the information visible and non-visible about a player house, items, etc, and the main one that handles the chunks that only have visible information about all the houses within a chunk(imagine the chunk as the key, and the houses as the items of an array).
The issue is that the old fashion way of thinking of each player/user id as a house doesn’t work because since the houses are stored forever on the map, they must be loaded quickly and thus in a “chunk-like” way.
you can also consider a chunk to be a neighborhood for this occasion, since many players live in it, each with their own house they can create/delete and modify.
to stop overwriting data for each chunk, you would now need to keep a copy of each chunk’s data consistent with all the other servers data.
we dont need to mention how bad that would be when the game has dozens of servers and 10’s of players manipulating their houses at a time
what you could try to do is have only 1 server that saves the data every 20 or so minutes, then when that server closes down, the servers have a death brawl to decide the next hunger king.
when a server starts up it’d ask if there was a chunksaver, and if there was no response it’d assign itself the job and when other servers started themselves up they’d see there was already one and they’d say ‘im not gonna save any chunks, but i’ll send any changes i have and i’ll replicate changes from other servers.’
that chunksaver would save all of the chunks every 20 or so minutes. or if your feeling dicey, you can have it only save when there arent any other servers to take the mantle.
you may think that having it save every 20 minutes makes the servers out of sync, but you can simply have the new severs ask for the data that the chunk saver has instead of loading the datastore. and only load the datastore when there are no other servers online
there are probably a couple of flaws in this idea, but i cant see anything wrong with it as im writing and i’ll edit it as i see any flaws.
Yes, they need to replicate across all the active servers when changes are made.
The chunks exist as information in an active server if at least a player is near them or has a house there and they load in as physical parts on the client if they approach them(for example if they’re 2 chunks distance away or less). That way multiple clients can’t cause lag spikes to each other(I could also use streaming enabled for this, I might consider it).
That sounds like a good idea, basically, we could consider each server a client in a serverless environment, and basically consider one the main server according to specific criteria and make a system that if something happens to the main server, it picks another.
However, there’re flaws even with this, for example, if all servers shut down at once(if Roblox is having issues) or if the main server is killed in such a way that it doesn’t have time to notify other servers and they basically have to detect it themselves.
that’s why we have it save every 20 or so minutes, and when compared to other options and the fact that no matter the solution there’s going to be a issue this is most likely the best solution.
and while the servers shutting down at once is a problem, its not as if all data is lost (unless it is) and bindtoclose can wait up to 30 seconds before the server fully shuts down, which is more then enough time to send that data.
you could also implement a system where if robloxs api services are offline, you simply shutdown the game.
as for the shutting down all at once, you can simply make a variable and set that variable to true when bind to close is imminent and make it so it wont send if that variable is set to true. allowing for the main server to see that all games are going to close.
however this isnt to mention the fact that we’ve never heard a story where all of the servers of a roblox game shut down at the same time, without manipulation from the developer themselves
You’ve made a pretty tough situation with this, but we can work it out.
Alex’s idea is OK, however lets assume the worst case scenario happens and the main data saving server crashes and saves absolutely no data. From there, you’ve lost possibly hundreds or maybe even thousands of save requests. Not to mention the pressure it would put on that server.
What I’d do is this: Keep track of what chunks are loaded globally, then if two or more servers load the same chunk, lock saving to one server. The other servers then relay anything that’s changed about that chunk to the data saving server. If the data saving server crashes, the other servers still have that data loaded, and they can decide to promote a different server as the data handling server.
Realistically, the only issue with this is that builds can be overwritten by other people’s builds, so my idea is to use something similar to ForeverHD’s infinite player server to maybe show “ghosts” on that server also changing the chunk, building and such (which would be quite funny and could lead to some in game lore).
In the end, because it’s decentralised, there will always be arguments between servers and data overwritten, that’s just the nature of systems like that. It won’t be easy to implement, so good luck.
I was thinking of implementing a system like that to broadcast player locations and chat messages to make the entire thing a living hell, but let’s not go out of topic with that one.
the other servers are also taking in the replication requests, not just the data saving server. so even if the data saving server crashes instantly, the other servers still have a very similar if not the same chunk data.
My point is that between the server being recognized as having crashed (which could take a while), many servers could have already sent many save requests, and MessagingService doesn’t store requests. So in between the time that the central data server dies, and the next one is chosen, a lot of data could be lost.
Edit: Unless, of course, you use MemoryStoreService, which isn’t a great idea, but it could probably work.
Not on loaded chunks. If a loaded chunk updated, it would look to players like someone invisible was building there. Unloaded chunks, sure, that’s fine, but it’s not good on loaded chunks. Either way, I did say that it will put a lot of pressure onto the data saving server (and also possibly MessagingService, let me check).
I’m personally thinking of how Minecraft does things, as this seems suspiciously similar to it. Anyway, this is a bit offtopic, so here’s my final thoughts @NyrionDev:
After looking into MemoryStoreService, it’s probably your best bet for this kind of system. It combines the centrality of Alex’s idea but removes the downsides, in this case server crashes and pressure. You can use it to both keep track of loaded chunks, and then use MessagingService to relay data between servers with those loaded chunks, and then the last server to unload the chunk gets to save the data.
Unfortunately, it’s probably your only solution to this. Your entire game based off your descriptions is a living hell (at least on the backend).
I have one last idea: split groups of chunks into places, and run only one instance of a server each. Teleport players around as they go around the map. If you need more, use the place creation API or something to create more places ad hoc. Probably less of a nightmare than what you’re trying to achieve.