That is actually what I already do. I even use the same algorithm for compression. It is around 13 MB uncompressed and 1-2 MB after compression. The issue is that there are over 4 million different values required for a chunk, as the table size is 64 x 513 x 64, so 1-2 MB is likely the lowest possible.
Also, I am not sure why a compressed JSON would be incompatible with the datastore. I could be wrong but I believe it is just a string, not some data type like Color3 which is lost when being put in to the datastore.
Can you send me data of an uncompressed chunk? Or a bit of it? I want to see how you store the uncompressed data and see if itâs possible to decrease the data count. Also for JSON, I meant by JSON alone is completely worse in terms of data compression than what I found during my research. JSON is an algorithm that compresses things into strings, it can only compress things that are numbers, strings, tables, and booleans. During my tests on JSON, it would sometimes lose data, Iâm not sure exactly why thatâs the case, but I ended up making a table-to-string system. Another thing is that JSON is compatible with Datastore due to it being a string converter. The way LZW compression works makes it not compatible with Datastore due to its non-ASCII elements, which is why I suggest you convert it to Base64.
While researching I found a way to sort of âbypassâ Robloxâs datastore request limit. The only issue with this is that you would need an external database, which can end up being costly. Using HTTPService, you are able to request and store data in the database. Instead of the 25 (I believe) requests per minute Roblox has, you will have up to 500 per minute. Hereâs a link to the article about HTTPService if youâd like.
As for the full uncompressed data, it would be way too long to put here, but I will show the general structure of it.
local materials = {
-- X axis
[1] = {
-- Y axis
[1] = {
-- Z axis
[1] = Enum.Material.Grass,
...
[64] = Enum.Material.Sand
},
...
[513] = {...}
},
...
[64] = {...}
}
local occupancies = {...} -- Same as materials but with numbers such as 0.125
^ This is then converted to JSON, though I havenât really tested it for actual use, so it most likely sets all Enum values to nil.
And as for the bypass, I wouldnât really call it a bypass. Itâs more of a different solution. I already know how to do it though and I already mentioned this was the alternative solution in case there wasnât another way.
As for what I can see in your way of storing data, there are two things that I would change. The first one which will significantly reduce data is changing the Enum.Material as a number or ID. Doing this will use less data as youâre using 1-2 characters instead of 20+ characters. When you decompress it, having a dictionary with each material and then finding the material using the ID should help reduce data. Another one which you could do is changing the table keys from X,Y,Z to X,Z,Y or any other way, this can reduce data a little bit, but not by much. In my testing, changing Y and Z coordinates to Z and Y reduced data by about 3-8%.
Edit: After doing a little bit of research, Iâm wondering how you used occupancies as well. Does it repeat the value of the same material? If so, try to also put it in the dictionary and retrieve it by the ID. Hopefully, this helps with your game!
Currently planning on just using an open source serializer with compression to handle it but I will consider that. Though, switching XYZ with XZY would be very difficult, as this would require the table to be reassembled before putting it into Terrain:WriteVoxels() as it only takes tables in XYZ format.
Occupancies are what Roblox terrain uses to determine how much space it takes up. Something with an occupancy of 0 is just empty, while something with an occupancy of 1 is a completely full voxel.