Hi! I am working on a game that is currently public called “World Builder” and have run into an issue where worlds with large amounts of blocks can no longer save because the size of their data has become too large.
How can I make the size of the data saved smaller/compress it to fit a lot more data? Currently all of the world’s data (including information such as the name, blocks, etc) is saved under one datastore, should I also separate the blocks data from the rest of it? The issues with saving start appearing in the world when it reaches somewhere around 750+ blocks
Currently the structure of my data for a world and a block look like these:
Block:
{
GUID = E9B30BD5-6D77-4525-B2E4-D9B2B2F61EE2;
ID = 1;
Position = {x,y,z};
Rotation = {x,y,z};
Size = {x,y,z};
Color = {r,g,b};
Config = {CanCollide=true,CastShadow=true,etc...};
Connections = {Inputs={},Outputs={}};
}
You are going to need to compress the data. This can be done a few ways. One easy way would be to make your entire data structure an array. Dictionaries take more space to save. You should also assign codenames to everything if you must use a datastore.
Example: Config = {C = 1, S = 1, ...} where C = CanCollide, S = CastShadow and 1 being true.
You could also use “references”. If two Block objects have the same properties then set the block that is farther in the array to have its value reference back to the property for the earliest version.
Lets say the block below is the 10th block.
{
"GE9B30BD5-6D77-4525-B2E4-D9B2B2F61EE2", -- GUID
10, -- ID
{x, y, z} -- Position
{x, y, z} -- Rotation
1 -- Size
5 -- Color
2, -- Config
{{}, {}} -- Connections
}
In this case it would have the same Size as Blocks[1], the same Color as Blocks[5] and the same Config as Blocks[2]
The same goes for World objects. Make them into arrays, and essentially do the same “reference back” process.
You can serialize data too. This is more complicated but if you wanted to, you could even make your own binary file format (or even use file compression) to store things in even less space. I used to store player-drawn skins for a minecraft knockoff by storing a sequence of 3 binary integers for every pixel (3 bytes per pixel) which was far smaller than storing a color integer, the RGB values, or an array of RGB values joined together.
I tried out what you suggested and combined it with the code from this post and it really did a lot, I found that the size was more than halved using this method but I want to know, what do I do about the Config table? I cannot hard-code it like I can everything else because the Config is unique for each type of block.
It is, like I said before, more complicated. Serialization comes in levels, and the process is that you end up converting existing data into something more compact. The more compact you make it, however, the more time you have to spend trying to read and write the data and the more complex it gets. Compare for example, sending a small compressed .png file to someone, then compare that to sending them raw pixel colors as an array of RGB. It is far easier to read the pixel colors, but the file size is gigantic in comparison.
Alright well right now I am considering using the other method I have made and just saving blocks across tons of separate keys in a ordered datastore unique to each world.
The BitBuffer module (tutorial) lets your fairly easily squeeze every last bit out of data stores. You get to make choices for how many bits it takes to save every type of data, so e.g. bools will actually just take the one bit they need, and you can decide how much span vs precision you want for positions, sizes, transparency, etc. You can get huge savings! And using that text compression thing on the base64 string afterwards would probably compress it a bit more.
It gets pretty technical and fiddly, and makes your saving system a lot more complex. But if you really need to space, then this is the way I recommend.
at the moment I plan to use a compression function I made that appears to be capable of allowing 16k blocks to be stored in one key and am going to save them to new keys once the last key gets full.
Storing tables is not very efficient in datastores. As others pointed out, you can reduce key name lengths. Another option is to store a string instead.
If you want to compress the string even more, you can use a huffman tree, as explained in the linked post. This will work better, the less block types you have in your game.
You could consider storing changes made instead of the entire world.
There aren’t any changes to be stored to begin with, the entire world is made by the player from scratch and nothing is procedural, I have a lot of block types so that wouldn’t work well either based on what you said. I’ve already figured out a solution that will likely work out well (compressing the data and saving them to multiple keys)
A Huffman tree always compresses strings, regardless of how many character types (block types). It’s especially effective if there are few of those types.