What is the fastest / most efficient way to compress a stringvalue?

I am trying to save as many characters as i possibly can, but how could i decrease the size of this string?
People can build massive creations in my game with over 10.000 blocks.

104914,38,12,2:1:t:2:1:3:f:4:t:5:f:6:30:;1;M;2;N

You could try converting the numbers to base 32, but without knowledge of how this string represents a item, I cant help you much more unless you provided more information.

Why do you need so many characters for one item? What do they represent? I can see that the first 3 are position, but what are all the letters for? You canā€™t just compress a string.

1 Like

104914(position converted to number) , 38(rotation converted to number) , 12(BlockID) , (Configs / etc) 2:1:t:2:1:3:f:4:t:5:f:6:30:;1;M;2;N

But what do all those configurations represent? There is most likely a way to store them better, if you tell us.

Otherwise, just converting the numbers into base 32 or 64 or whatever would be the only solution.

The keys used for the blocks, like you can throttle up with M and down with N and the speed / height / etc

So then your only solution might be to make a max of the number of ā€œspecialā€ items you can place. Or make one generic keycode to control the whole thing. Otherwise, you cant really compress it further. What is your use case for having keybinds for every object for 10,000 of them?

so there isnā€™t a compressing method for my string? and the keybinds dont show up for every single block

You cant compress a string into a string. For numbers, you can use a different base. For rotation, you can use a set preset. For colors, use can use hexadecimal (a different base). For just strings, you canā€™t really do so much. Iā€™m confused as to what you want us to do.

This probably isnā€™t very helpful to you because itā€™s not answering your question, but, donā€™t you think it would be good to start thinking about how to overflow into other DataStores now? I donā€™t faithfully think you can reduce this further; and definitely not with any speed or efficiency. As it is, your current implementation is unknown.

Youā€™ve already compressed your string by quite an amazing amount, so itā€™d be quite difficult to reduce this even further without losing accuracy or information. Data doesnā€™t exactly have to be readable but you still need a way to convert easily to and from storage formats and readable formats so you can actually work with them. The rest comes down to how you limit building absurdly large structures in your games (plot size and content considered) and how you handle data.

Overflowing into secondary keys isnā€™t a bad idea to start considering at this point, but be sure that youā€™ll have a way to know if thereā€™s overflow and how to fetch from secondary places as well.

2 Likes

I have figured out that i could link strings that match without the position to other already saved strings in the table.
This decreases the savedata by a lot.

If anyone knows more ways to decrease the savedata, please let me know.

1 Like

In case your data-fields for the ā€œconfigs/ etc.ā€ should always be there, then you could avoid storing fields-with-values that are equal to the gameā€™s default values.

For example, from your cryptic storage-string 2:1:t:2:1:3:f:4:t:5:f:6:30:;1;M;2;N

If that 1:t is supposed to represent; field-#1 with value true, and this field-#1 in the game only has valueset (true, false), and the gameā€™s default value for it is true, then there would be no need to persist it into storage.

When retrieving from storage, and parsing the storage-string, the parser must know, that field-#1 should be set to true, when field-#1 is not found in the storage-string.

This could be expanded to several fields, with a study of what value is the most frequently used for a field, and use that value as the fieldā€™s ā€˜default valueā€™.

So in case the gameā€™s ā€˜default valuesā€™ for these three fields are specified as:
field-#1 = true
field-#3 = false
field-#6 = 30
Then the ā€œconfigs / etc.ā€ of the storage-string could be reduced to: 2:2:1:4:t:5:f:;1;M;2;N

Only when a fieldā€™s value deviates from its ā€˜default valueā€™, it must be written to the storage-string: 2:2:1:3:t:4:t:5:f:6:25:;1;M;2;N

Another idea could be, if those #:t: and #:f: entries do represent boolean-typed values (true, false) for a field-#, could be to simply not store fields with value false, and represent a true value with just #::

A further idea, looking at all those delimiters, would be to reduce those too. If there is never going to be more data-fields than a maximum of 62 (26+26+10), then that {field-number}{delimiter} format could be replaced with a single symbol from the set [a-zA-Z0-9].

So
2:1:t:2:1:3:f:4:t:5:f:6:30:;1;M;2;N
could be reduced to
2:at:b1:cf:dt:ef:f30:;1;M;2;N
or when also using the knowledge of ā€˜default valueā€™ and ā€˜booleanā€™-types, then to
2:b1:d:;1;M;2;N

1 Like

Lets say someone dug a cave of 2x2x1000 on your game, so the terrain Material turned into air. You can make a database of your materials, like {Air=0,WoodPlanks=1,Rock=2,etc}, so instead of writing the material you just write one number for it, saving some bytes of datastore. Then, you set the corner of this cave you dug as pos1, and the opposite corner as pos2 to reference how much blocks you need to place, so Instead of saving 4000 dig positions, you just need to save two. You can compress it even more by making a dictionary like this: {Blocks={0={232={21={313,532,252}}}}
0 = "Air, 232 being x coordinate, 21 the y coordinate, and 313,532,252 as the z coordinates of three pos1 of your block, you go even further to place the second coordinates. That if dictionaryā€™s can handle that much subdivisions, which I think it does.