I'm so EXCITED about my compression

[ColdSmoke]

One of my testers decided to push his plot to the max. At 32,000 Part count, this entire plot is being saved exactly how it is at only .70% of the 260000 character limit of a datastore key

Now this is a lot of repeated information which makes it very very easy to compress, a user actually using a variety of pieces should have a larger file size, but this is incredible. And there’s no lag, due to me making plots 100% client sided. My game could handle a large playercount, all doing this individually with almost no tax on the server

I’m just very excited about this and I’d like to share
(edit: More accurate example below)

[EmeraldSlash (Rio)]

Wo cool

also I thought the title said depression

[rbx_lua (Matthew)]

@ColdSmoke

260000 character limit of a datastore key

I’ve actually reached this before with only 2,000 values (one per grid cell) before, so I’d be very curious to hear how you created this.

[ColdSmoke]

I actually had that issue a lot in classic homebuilder

Once I release I’ll make a detailed tutorial on how I achieve such good compression + a general datastore tutorial since it seems a lot of devs could really use it. But basically I have a multi-dimensional array, where shared values are repeated the minimum possible, and more unique values are saved individually, then it’s JSONEncoded and ran through LZW Compression.

The entire method is lossless.

[Maxx_J]

amazing work!!!

[rbx_lua (Matthew)]

I also implemented a multi-dimensional array but I had not considered reducing repeating values nor would I be sure how to go about doing that. The JSON and LZW Compression sounds interesting, I’m going to look into that.

Fortunately I have never had any issues with the data stores losing data, I’ve just prevented them from creating so much data that it could fail to save in any case.

[ColdSmoke]

Oh and I forgot, it’s going to become even smaller once I localize the position values to the plot position.

@RBX_Lua JSonEncode is easy, it just converts the array to a string, and then I got the LZW compression from @1waffle1. I know he has… concerns about its usage but I’ve found in my case that it’s been totally reliable.

[1waffle1 (Waffle)]

My concerns are really more about my hypothetical uses than yours. Compression algorithms have no guarantees of making data fit inside constraints. You’re just looking for more room, and you’re compressing data that’s pretty ordered, so it will do a good job of representing that order in less space.

[tyridge77]

Sometimes I watch sad movies and get really compressed

On a serious note this looks awesome ColdSmoke! I need to look into LZW for this sort of thing.

[EmeraldSlash (Rio)]

I can relate.

Sometimes the crushing feeling is too much.

[Aesthetical]

Better patent it and create a company called Pied Piper now

[reteach]

Thats pretty freakin cool

[ZarsBranchkin]

I’m curious about the PostAsync compress argument. It says, that when enabled, your request will be gzipped. Gzip will dramatically reduce the sent data size and will optimize repeated strings, so it’s perfect for this kind of situation.

The thing I’m uncertain about is whether roblox will gzip your string first and then check if it’s <1024KB or will it check the string size, before gzipping? If the prior, then there isn’t really an reason to create your own compression code, but I’m not certain about how exactly roblox handles it.

Reference: Documentation - Roblox Creator Hub

[ColdSmoke]

My game homebuilder constantly ran into this error before I was using LZW compression, multiple people would get this issue daily.
error while saving: 105: Serialized value converted byte size exceeds max size 64*1024 bytes.
After I implemented LZW, there hasn’t been a single instance of that error being thrown. I log all saving errors on my discord server using webhook.

[Fractality_alt]

Nice! I wonder if you could pack it down even smaller by running RLE first.

[ColdSmoke]

You just gave me an Idea, I know how I could optimize this WAY WAY more, for RLE
If I broke up my Vec3 positions into individual lists where the position on that list kept correspondence with (for instance) the Y and Z lists, I could WAY shorten the amount of space that positions take up simply by grouping items which are “snapped” together

So this is a 2x3 of walls

instead of saving like it currently does which would look like this

[1] = {["X"] = 2,["Y"] = 0,["Z"] = 1,},
[2] = {["X"] = 2,["Y"] = 0,["Z"] = 2,},
[3] = {["X"] = 2,["Y"] = 0,["Z"] = 3,},
[4] = {["X"] = 2,["Y"] = 1,["Z"] = 1,},
[5] = {["X"] = 2,["Y"] = 1,["Z"] = 2,},
[6] = {["X"] = 2,["Y"] = 1,["Z"] = 3,},

I could save it like this

["X"] = "2*6",
["Y"] = "0*3 1*3",
["Z"] = "(1,3)*2",

Right now I’m not doing anything special for the Vec3s (aside from rounding them to the nearest 1/10th), mostly because the multi-dimensional approach wouldn’t help that much since I can’t predict which axis players are going to prefer(although I do have it set up so it could detect the best compression and method and save it like that) but hmmm yes

Thanks for the idea!

[zeuxcg]

Would there be any interest in us exposing some compression utilities, like deflate and/or lz4? We have some seriously fast compressors available in the engine and we use them for various tasks, like compressing file data, network traffic, etc.

[ColdSmoke]

Yes please

[Velibor]

Always

[davness]

I’m in.