How to compress data store?

[0786ideal (ideal)]

Hello, i have this problem, where in my game there might be some data like invidual item data and stuff like that, it might take some space in data stores and i really want to compress it

Problem is that idk how to reduce amount of characters without making it unredable, for now it’s example of a item stack

Inventory[2] = {
    Item = 210,
    Amount = 57,
    Attachement1 = "Lamp"
    Attachement2 = "Plate"
    Attachement3 = "Rail"
}

As you can see, those strings have a lot of characters, how can i compress them?

[happya_x (Happya)]

you could store an array and keep track of the keys separately

if you want to compress even more maybe you could put the array data into a buffer, and then store to the datastore as buffer.tostring()

inventory = {
  210,57,"Lamp","Plate","Rail",
}

--remember keys are item, amount, attachment1... and so on

you could also map the item names in a dictionary to save characters on strings

inventory = {
  210,57,"i1", "i2", "i3"
}

local keyToItem = {
 ["i1"] = "Lamp"
  --and so on
}

[0786ideal (ideal)]

Sadly those arguments are dynamic, and i need to keep track of them, i store item names as numbers for now

[VegetationBush (FartedWayTooHard)]

You can encode it using HTTPService:JSONEncode(), pass the string into a string compressor, then save that compressed string.

To load the compressed data, load the string from datastore, decompress the string, then use HTTPService:JSONDecode() to decode the string.

[fork_lifted]

You could try GitHub - cipharius/msgpack-luau: A pure MessagePack binary serialization format implementation in Luau
Make sure to use the “utf8” variant of the functions (ex: utf8Encode) if you plan to use the data in Datastores

(its not directly “compression” but it serializes your table and makes it smaller)

[0786ideal (ideal)]

How to create string compressor?

[VegetationBush (FartedWayTooHard)]

You can create one if you know the algorithm. However, people usually use an existing one. There are lots of them you can use if you search it up on google.

[0786ideal (ideal)]

I’ve searched a bit and they are mostly used for test, not single words like Item, Attachement ect.

Because i don’t want to loose redability for my scripts over data store compression, idk if there is other way to do it

[VegetationBush (FartedWayTooHard)]

I’m not sure you quite understand what compression is.

Compression is where you take a piece of data, and then change it in certain ways that make it take up less space, and in this case, without losing information in the process.

The string returned from the compression will be unreadable, however, that’s why you decompress the string to make it readable from scripts again.

[Dragon_Trance]

If what you’re looking for is string compression, here’s a quote taken from stackoverflow:

First, hopefully it’s clear that there does not exist any lossless compression algorithm that can take an arbitrary string of length n and always compress it to a unique, shorter string. That’s a fact of math.

That being said, there’s some popular algorithms that work pretty well:

Huffman Encoding: fairly beginner-friendly and possible to implement yourself. The basic idea is to map more common characters to shorter binary strings and less common ones to longer binary strings, and then package that with a map that tells you how to decode the resultant bitstring. The downside is the extra space you need to store the decoding instructions

Lempel-Ziv: I’ve never implemented this myself, but it’s the basis for a lot of common file formats we know today, like GIFs. There should be libraries out there for this.

My approach for “compression” was to simply take away key-value pairs and introduce a slot/versioning system (inspired by SS13 code)

Essentially, to save data, I feed a table of “parsed” data into a function. The function is responsible of translating all of the key-value pairs into slots/indexes of a new table.

After that gets converted, it gets saved to the datastore service responsible for keeping track of the player’s data. When the player joins, it’s upgraded to the latest version (since at any point, data can be changed and moved around during development), and then converted to a “parsed” table.
This is how I implemented it though. I don’t use compression.

But, unless you have an incredibly large amount of data, compression isn’t exactly needed. The most amount of data you can save for any data is 4 megabytes, which is plenty more than enough space to save data for a player.

[0786ideal (ideal)]

I know, but my problem is that most compression algorithms are used to reduce number of data inside long strings with repeating words, letter and ect. my strings are not that long, they are 10-12 bytes, but if there will be a lot of them, they’ll eat up data store’s space

[0786ideal (ideal)]

Thx, so do you think if i used few kilobytes i should be fine?

[Dragon_Trance]

To put 4 MB into perspective…

1 kb is 1,024 characters.
4 kb is 4,096 characters
16 kb is 16,384 characters…

1 MB of data holds 1,024 kilobytes. 1,024 kilobytes holds exactly 1,048,576 bytes
4 MB of data holds 4,096 kilobytes, which is 4,194,304 bytes

Whether or not you think “a few kilobytes” is fine, that’s up to you, but it’s hard to get a single kilobyte of data for a player unless you have LOTS of things to save. It depends on what you’re doing to say how much is too much