-
What do you want to achieve?
I need a way to save and load data more rapidly than the datastore service allows, though mainly loading, as I only need to save chunks that have been modified. -
What is the issue? Include screenshots / videos if possible!
Currently, if a single player were to walk diagonally across 8 chunks in one minute with a render distance of 2 chunks (it is done in a circle but for simplicity’s sake, we will use a 5x5 square), the game would be unable to load any other data for the rest of the minute.
And this is even worse as more players join the game. With 100 players, they could only travel around 2 chunks going horizontally. Additionally, just spawning in would wipe out a whole 25 requests from loading the terrain around the player, so if 10 players were to join an empty server in one minute, the game would just fail to load anything else, not even including other datastore requests that might be made such as loading their data.
The chunks are quite large, around 256 studs horizontally and around 1024 vertically, which was intentional because of this issue, though they do take a moderate amount of processing power to generate. -
What solutions have you tried so far?
Admittedly, I do have a solution, though it requires the use of an outside tool and I would prefer to keep the data on Roblox, as it’s faster and easier to use the datastore service. So, is there any way to actually use the datastore service in this scenario?
Have you tried using a Module called Datastore2? It’s a more advanced way to datastore. Here’s the Module and Documentation here. Module: Release DataStore2 v1.3.0 · Kampfkarren/Roblox · GitHub Documentation: How to use DataStore2 - Data Store caching and data loss prevention
It sounds like it could be useful, but I don’t think it would allow for a higher request limit (though, feel free to correct me if I’m wrong). The issue is more of a Roblox limitation than something I don’t know how to code.
Roblox Datastore can hold up to 4MB of data (which is kind of a lot) If you were to go past that limit you would have to be coding a rocket ship in roblox lol
The issue isn’t a data limit, it’s the request limit. You read the post, right?
The game uses a chunk based system and if you were to walk around 8 chunks diagonally in one minute with a 5x5 chunk render distance, it would hit the data store request limit of 70 requests per minute (assuming you were the only player in game). This scales up very badly and can even hit the request limit at only 2 chunks horizontally in larger servers.
Basically, just imagine a 5x5 grid around the player. When the player moves into another square of the grid, it has to load more chunks in, but Roblox won’t allow more than 60 + 10p (p being the amount of players in the game) chunks to be loaded in per minute.
Apolgizes for the misread, you could pre-load a bunch of chunks, so that it doesn’t reach the request limit.
Pre-loading is still loading, there isn’t nearly enough on the request limit and many players will likely be exploring the world. Pre-loading would actually be even more likely to hit the request limit as it would be loading chunks that players might not even enter.
You should look into it more and let me know if you get anywhere. Once you get code I’ll definitely be happy to help if there is any problems. I’m not the guy for ideas I’m more of a worker lol
I have coded quite a bit already. The game actually fully works except for saving and loading chunks. The issue is the one that I already mentioned, the datastore request limit.
To clear things up, you’re calling GetAsync for that specific chunk that they enter… everytime?
GetAsync would be required once for every chunk, so it would be about twenty five calls to load just a 5x5 chunk area around them. Chunks are currently never unloaded though, so if you were to walk in a 2x2 chunk circle then you would only load in 36 (6x6) chunks no matter how many times you crossed chunk borders.
Unfortunately, loading a giant 64 x 513 x 64 array of terrain data takes a lot of space (at least until I improve the compression method), so putting multiple chunks into a single call is not an option as far as I know.
I don’t see the issue of putting multiple chunks in a single call.
DataStore has its storage limits, sure, but I doubt you’re going to reach it in this case.
I did actually test it and even a single chunk is over the data limit uncompressed. The data limit is 4 MB or 4,000,000 bytes. A single chunk has over 4,000,000 values (64 x 513 x 64 then x2 because it’s both materials and occupancies), and even a single value tends to be multiple bytes long. It can be compressed down to around only 1-2 million bytes but this isn’t small enough to store a meaningful amount of chunks.
I think you’re overcomplicating it if you’re just storing studs.
You could store a value of 4,000,000 as just 4M in this case (If I’m correct about what you’re doing, that is.)
I’m not sure you know what I’m talking about. This isn’t studs, this is the built-in smooth terrain system. It is based on voxels and to save/load terrain, you require 2 tables, one with the materials and one with the occupancies for every single voxel. One chunk in my game is 64 x 513 x 64 voxels in size (multiply each value by 4 to get it in studs) and to save or load this requires 2 tables of the same dimensions sent to Terrain:WriteVoxels(). Additionally, there is no “value of 4,000,000”, there are 4,000,000 different values that are absolutely required to load a chunk of terrain. The only way to downsize this would be to improve the compression of the data, which I will do eventually, though it would take a rather long time to code and I am busy with some more important things at the moment.
What does your chunk data look like? I’m not sure how and why you need a chunk-based game, but is it procedurally generated or is it modified by the player in some sort?
It is procedurally generated as mentioned before, though players can modify it. The chunk data is the same as the data received by Terrain:ReadVoxels() and used by Terrain:WriteVoxels().
From what it sounds like to me you are trying to find a way to surpass the datastore’s rate limit, which to my knowledge isn’t something you can mess with. Could you try making the chunks smaller, thereby allowing them to load faster and emptying the queue quicker so you don’t end up throttling the datastore?
If that doesn’t work or isn’t an option then you should probably just go with your outside-source option.
Bottom line is, you can’t change or increase Roblox’s request limit.
As 1Urge said, you cant surpass the data store limitations. What id suggest doing is “mixing” chunk data together. What I mean by this is that each key in your datastore has more than one chunk. As you said above, each chunk uses over half the amount of data in each key, making it unable to add multiple chunk data in a key. What I would do is add a compression algorithm. Currently, I’m making a voxel sandbox game, and my compressing algorithm changes a table into a string and compresses the string using the LZW algorithm. Here’s a link to read about the algorithm.
Link to 1waffle1 LZW Algorithm in Roblox
link
This got me by much farther than JSON encoding and other algorithms did so far, in my tests, a chunk that is 16 x 256 x 16, would use around 23KB, using LZW, I was able to get it under 2KB. This algorithm, however, changes the data into a format that Datastore does not support, I suggest converting it into Base64 for that. Yet this won’t bypass Roblox’s requesting limit, this will definitely compress Chunk data and let you fit more chunks into each request.
Making chunks smaller would be worse, as more would have to be loaded in more frequently, and this issue only gets exponentially worse the more you downsize them. Loading is not a problem, as Roblox turns out to be incredibly efficient at loading fairly large amounts of terrain data.
I will most likely be using the outside option, since it doesn’t seem to have a rate limit, but instead more of a different kind of limit.