How would I compress thousands of blocks in json data into smaller data

Help and Feedback Scripting Support
, ,
#1

I am making a block game and the blocks placed by players will be saved into Datastore.
the blocks format are stored like this:

{
  [1, 1, 1] = 1,
  [1, 1, 2] = 1,
  [1, 1, 3] = 1
  -- and 1,000 more
}

The value is the block’s position, and the key is the block’s type.

The problem is that I’m storing thousands of blocks, and that’s a really lot if you would imagine.
So obviously I would want to find a way to compress it.

I already had come up with an algorithm here, which is to pack the current key and the previous key into one and calculate the incrementing index of the position into one.
However, the block data is not ordered and randomized. And I do not know how would I code and achieve my algorithm.

Is there any algorithm method I could use to compress them or do just I have to make my own one using a formula?

QuadaStore - The ROBLOX Datastore Solution
#2

You might be interested in reading about how roblox compresses their voxel data for their terrain: zeux.io - Voxel terrain: storage

You may also be inspired by googling “voxel storage techniques”

#3

Thanks!
I’ll take a look at that and try my best to implement the algorithm based on the techniques.

#4

No way it actually works!

I have a block data of 234,000 Bytes and compressed it using zlib lua,
the output was 297 Bytes which is a 99.87% decrease!

Thank you for mentioning the keyword to search!

#5

Careful trying to save that, roblox doesn’t let you store proper binary in data stores

So you’ll probably have to encode the string as base64 or something again

1 Like
#6

Alright, I’ll try that thanks!

Probably should not increase the string size anymore since I updated how I store my blocks.
It’s now 97 bytes.