How do I/Is it possible to save huge maps like minecraft?

I have a seed world generator and I want to save the map changes on datastores (ex: i mined a block at x position, save that change and apply when loading the chunk again)

The problem is that datastores seem to have a maximum of 4 mbs per key. A voxel takes 2bytes to save, not mentioning the coordinates, allowing for some millions of changes. But If people mine/build as much as when they play minecraft, I think it could reach that number easily.

Maybe I could save on more than one key, but I don’t know if that’s problematic, because it would be way more data to save and I think they have a reason to limit storage. I also thought of returning a string for the player to copy and save for later use to load the game but I think GUI objects don’t hold that much text.

It’s possible, sure. You could use string.pack to turn binary data as a string and save that string.

What binary data? That’s up to you. But the simplest (like a on or off voxel) would just be NxMxL bits, one per voxel, that you could compress with lz77 or something.

You could read about how ROBLOX handled storing the data for their terrain (and how it differs between long storage and in-memory storage) in this blog: zeux.io - Voxel terrain: storage

Maybe it will inspire you.

Even with smart compressing and packing you still may need to split the data into multiple strings or even keys.

Edit: also your idea about only saving changes is a good one, but maybe see if you can avoid it. It will become a massive headache in the future, for example if you change your generation code you now need to make sure that you store what version you originally generated it with so you can use that old version (with all its bugs) to recreate the world on rejoining.

1 Like

Oh I was just remembering about a video I saw where a guy stored columns of blocks with the same dimension into one big part, saving lots of data waste. Its the same as you mentioned here and in the link, and I think I might be able to do it. thanks a lot!

Thanks to your help, I think i might be able to reduce data consumption on my datastores by thousands of times. An example of a 2x3x4000 voxels tunnel dug by a player would occupy something like 407.898 bytes of data, which is more than 10% of the limit already. But with the compressed blocks and then the string, I think the same tunnel will occupy just 26 bytes, which is 15.688 times less data!