Apolgizes for the misread, you could pre-load a bunch of chunks, so that it doesn’t reach the request limit.
Pre-loading is still loading, there isn’t nearly enough on the request limit and many players will likely be exploring the world. Pre-loading would actually be even more likely to hit the request limit as it would be loading chunks that players might not even enter.
You should look into it more and let me know if you get anywhere. Once you get code I’ll definitely be happy to help if there is any problems. I’m not the guy for ideas I’m more of a worker lol
I have coded quite a bit already. The game actually fully works except for saving and loading chunks. The issue is the one that I already mentioned, the datastore request limit.
To clear things up, you’re calling GetAsync for that specific chunk that they enter… everytime?
GetAsync would be required once for every chunk, so it would be about twenty five calls to load just a 5x5 chunk area around them. Chunks are currently never unloaded though, so if you were to walk in a 2x2 chunk circle then you would only load in 36 (6x6) chunks no matter how many times you crossed chunk borders.
Unfortunately, loading a giant 64 x 513 x 64 array of terrain data takes a lot of space (at least until I improve the compression method), so putting multiple chunks into a single call is not an option as far as I know.
I don’t see the issue of putting multiple chunks in a single call.
DataStore has its storage limits, sure, but I doubt you’re going to reach it in this case.
I did actually test it and even a single chunk is over the data limit uncompressed. The data limit is 4 MB or 4,000,000 bytes. A single chunk has over 4,000,000 values (64 x 513 x 64 then x2 because it’s both materials and occupancies), and even a single value tends to be multiple bytes long. It can be compressed down to around only 1-2 million bytes but this isn’t small enough to store a meaningful amount of chunks.
I think you’re overcomplicating it if you’re just storing studs.
You could store a value of 4,000,000 as just 4M in this case (If I’m correct about what you’re doing, that is.)
I’m not sure you know what I’m talking about. This isn’t studs, this is the built-in smooth terrain system. It is based on voxels and to save/load terrain, you require 2 tables, one with the materials and one with the occupancies for every single voxel. One chunk in my game is 64 x 513 x 64 voxels in size (multiply each value by 4 to get it in studs) and to save or load this requires 2 tables of the same dimensions sent to Terrain:WriteVoxels(). Additionally, there is no “value of 4,000,000”, there are 4,000,000 different values that are absolutely required to load a chunk of terrain. The only way to downsize this would be to improve the compression of the data, which I will do eventually, though it would take a rather long time to code and I am busy with some more important things at the moment.
1 Like
What does your chunk data look like? I’m not sure how and why you need a chunk-based game, but is it procedurally generated or is it modified by the player in some sort?
It is procedurally generated as mentioned before, though players can modify it. The chunk data is the same as the data received by Terrain:ReadVoxels() and used by Terrain:WriteVoxels().
From what it sounds like to me you are trying to find a way to surpass the datastore’s rate limit, which to my knowledge isn’t something you can mess with. Could you try making the chunks smaller, thereby allowing them to load faster and emptying the queue quicker so you don’t end up throttling the datastore?
If that doesn’t work or isn’t an option then you should probably just go with your outside-source option.
Bottom line is, you can’t change or increase Roblox’s request limit.
As 1Urge said, you cant surpass the data store limitations. What id suggest doing is “mixing” chunk data together. What I mean by this is that each key in your datastore has more than one chunk. As you said above, each chunk uses over half the amount of data in each key, making it unable to add multiple chunk data in a key. What I would do is add a compression algorithm. Currently, I’m making a voxel sandbox game, and my compressing algorithm changes a table into a string and compresses the string using the LZW algorithm. Here’s a link to read about the algorithm.
Link to 1waffle1 LZW Algorithm in Roblox
link
This got me by much farther than JSON encoding and other algorithms did so far, in my tests, a chunk that is 16 x 256 x 16, would use around 23KB, using LZW, I was able to get it under 2KB. This algorithm, however, changes the data into a format that Datastore does not support, I suggest converting it into Base64 for that. Yet this won’t bypass Roblox’s requesting limit, this will definitely compress Chunk data and let you fit more chunks into each request.
Making chunks smaller would be worse, as more would have to be loaded in more frequently, and this issue only gets exponentially worse the more you downsize them. Loading is not a problem, as Roblox turns out to be incredibly efficient at loading fairly large amounts of terrain data.
I will most likely be using the outside option, since it doesn’t seem to have a rate limit, but instead more of a different kind of limit.
That is actually what I already do. I even use the same algorithm for compression. It is around 13 MB uncompressed and 1-2 MB after compression. The issue is that there are over 4 million different values required for a chunk, as the table size is 64 x 513 x 64, so 1-2 MB is likely the lowest possible.
Also, I am not sure why a compressed JSON would be incompatible with the datastore. I could be wrong but I believe it is just a string, not some data type like Color3 which is lost when being put in to the datastore.
Can you send me data of an uncompressed chunk? Or a bit of it? I want to see how you store the uncompressed data and see if it’s possible to decrease the data count. Also for JSON, I meant by JSON alone is completely worse in terms of data compression than what I found during my research. JSON is an algorithm that compresses things into strings, it can only compress things that are numbers, strings, tables, and booleans. During my tests on JSON, it would sometimes lose data, I’m not sure exactly why that’s the case, but I ended up making a table-to-string system. Another thing is that JSON is compatible with Datastore due to it being a string converter. The way LZW compression works makes it not compatible with Datastore due to its non-ASCII elements, which is why I suggest you convert it to Base64.
While researching I found a way to sort of “bypass” Roblox’s datastore request limit. The only issue with this is that you would need an external database, which can end up being costly. Using HTTPService, you are able to request and store data in the database. Instead of the 25 (I believe) requests per minute Roblox has, you will have up to 500 per minute. Here’s a link to the article about HTTPService if you’d like.
As for the full uncompressed data, it would be way too long to put here, but I will show the general structure of it.
local materials = {
-- X axis
[1] = {
-- Y axis
[1] = {
-- Z axis
[1] = Enum.Material.Grass,
...
[64] = Enum.Material.Sand
},
...
[513] = {...}
},
...
[64] = {...}
}
local occupancies = {...} -- Same as materials but with numbers such as 0.125
^ This is then converted to JSON, though I haven’t really tested it for actual use, so it most likely sets all Enum values to nil.
And as for the bypass, I wouldn’t really call it a bypass. It’s more of a different solution. I already know how to do it though and I already mentioned this was the alternative solution in case there wasn’t another way.
As for what I can see in your way of storing data, there are two things that I would change. The first one which will significantly reduce data is changing the Enum.Material as a number or ID. Doing this will use less data as you’re using 1-2 characters instead of 20+ characters. When you decompress it, having a dictionary with each material and then finding the material using the ID should help reduce data. Another one which you could do is changing the table keys from X,Y,Z to X,Z,Y or any other way, this can reduce data a little bit, but not by much. In my testing, changing Y and Z coordinates to Z and Y reduced data by about 3-8%.
Edit: After doing a little bit of research, I’m wondering how you used occupancies as well. Does it repeat the value of the same material? If so, try to also put it in the dictionary and retrieve it by the ID. Hopefully, this helps with your game!
Currently planning on just using an open source serializer with compression to handle it but I will consider that. Though, switching XYZ with XZY would be very difficult, as this would require the table to be reassembled before putting it into Terrain:WriteVoxels() as it only takes tables in XYZ format.
Occupancies are what Roblox terrain uses to determine how much space it takes up. Something with an occupancy of 0 is just empty, while something with an occupancy of 1 is a completely full voxel.