What's an efficient method of saving a lot of data?

Help and Feedback Scripting Support
#1

So currently, the way I’m saving unlocked abilities in my game is by creating tons of boolean values with the names of the the abilities stored in each player’s leaderstats. If the ability isn’t unlocked it’s set to false and the opposite if the player does have it unlocked.

Would it be possible for me to instead create a table of “Unlocked Abilities” and have all of the booleans stored in there. If so, how would I go about creating a table with boolean values and saving it?

Or is there a more optimal method to save large amounts of data for a player?

#2

just make the booleans in side the script?

#3

Apparently large arrays of boolean values can be compressed very efficiently if you know what to do:

When you save a table in a DataStore, it gets converted into a JSON string, so a boolean value literally turns into the word “true” or “false”, causing it to take up at least 5 bytes per value.

In that other post we discussed the possibility of storing an array of up to 32 booleans as a single integer. So with some optimization the storage space can go from 5 bytes per value to 8 values in one byte.

Of course, this is only really worth it when you are storing hundreds or thousands of booleans. Roblox gives you about 260,000 bytes per key.

2 Likes
#4

How many is a ton, can you give us an order of magnitude? I can’t imagine a game having a number of named abilities that is anything close to large by computer memory standards.

About the smallest you can get it in a DataStore with simple encoding would be to bit-pack groups of booleans into single-byte characters. While JSON technically has 94 such characters available, using base64 would be a good compromise, because it’s standardized and you can easily get copy-paste code to encode and decode it.

If you need the data even smaller, you have to consider compressing the binary data before you even store it, using standard techniques like run-length encoding, or Huffman coding (or some other entropy coding), to get the data smaller before the binary-to-text encoding.

That said, I was answering your question from a purely academic point of view. I wouldn’t recommend you actually try to achieve the smallest JSON string possible for an array of booleans, your time is better spent.

You can assign integer IDs to all your skills and just store them in a Lua array, which you can write it to a Datastore. It’s going to be bigger than a base64 string, but do you actually care? I’m guessing we’re not talking about 100k unlockable skills here.

Keep in mind that you only really have to store the IDs of either the unlocked skills, or the locked skills, so you only ever need to have up to 1/2 the IDs in your save data. If a player has unlocked more than half the skills, you save the list of ones that are still locked, otherwise you save the list of unlocked skill IDs. Then, you just need one more boolean to store whether the saved IDs are the locked or unlocked skills.

But you might not even need to do that, again, depends on how many skills there are and what your real concern is. Just an array that maps integer skill ID to a 1 or 0 for every skill is probably fine.

5 Likes
#5

I have a theory where you could take a bunch of boolvalues, convert them to binary, but them together to form a number, and now you are literally storing bools at 1 bit per boolean, then when you need to retrieve them, simply convert the number back into binary and use the bits.

for converting binary to number:
its actually as simple as using tonumber:
tonumber(“11010100011001”,2)

for converting number back into binary, i found this off Stack Overflow:

#6

I actually hadn’t even thought of the fact that I could just save the IDs of unlocked abilities and that I could also just use 0/1 Integer Values rather than booleans if I were to create arrays of data.

Thank you very much for the in-depth replies, it is very insightful for someone who is trying to learn more about Datastores and efficiency. I appreciate the it greatly @Blokav and @EmilyBendsSpace.

1 Like
#7

That is the first required step of what I suggested above, regarding using base64. But this alone does not get you 1 bit per boolean in datastore storage, because JSON stores strings. So, suppose you want to store 6 booleans, “111001”. You can still do tonumber(“111001”,2) and get 57, or you can do it arithmetically: 1*32 + 1*16 + 1*8 + 0*4 + 0*2 + 1*1 = 57

But… instead of storing 57, which requires two characters because in JSON it stores the string “57”, you can use a base 64 system where you use a different single character to represent each of the numbers from 0 to 63. In Base64, the value 57 is represented by just the character “5”. That’s half as many characters as it takes to store “57” in JSON as a string.

1 Like
#8

aah ok, that makes sense ///////