Data efficiency

[Misinformater (Manteca)]

I’m having problems with a project… So I came up with this questions:

1. What’s the most efficient way to save up keys? I want to store UserIds without taking up so much memory, should I use utf8?
2. How prioritized IncrementAsync and UpdateAsync is? I’m not sure which of them is better for saving data.
3. What’s the most efficient data compression method?

And some advices: (Them’re optionals, if you don’t feel comfortable with them don’t worry)

1. Don’t use keys: Them will take data, instead use tables as concepts everytime you want to load/save data.
2. Use JSON or any compression: I always use HttpService:JSONEncode()/Decode, this one is your choice.
3. Use faster methods: Use modules to save/load/modify data.

Thank you all, salutations and goodnight/day.

[7z99 (cody)]

First thing, this probably would fit better in #help-and-feedback:scripting-support

@Misinformater

What’s the most efficient way to save up keys? I want to store UserIds without taking up so much memory, should I use utf8?

Could you elaborate on what you mean by keys? Like DataStore keys? Those are limited to 50 characters and you can use new APIs like :ListKeysAsync/ListDataStoresAsync to get a list. Are you talking about metadata which is a list of user IDs?

@Misinformater

• How prioritized IncrementAsync and UpdateAsync is? I’m not sure which of them is better for saving data.

I’m not too sure as to what you mean. They are two different things and have different purposes, IncrementAsync is to, well, increment an integer that is saved to a certain key. UpdateAsync is to update some entry or to completely overwrite data inside of the data store. It’s usually used interchangeably with SetAsync. But I would say that UpdateAsync is the best way to go when factoring in things like getting up-to-date data.

@Misinformater

What’s the most efficient data compression method?

There are many data compression algorithms, there are multiple available on this forum as well. But the first step would be to optimize your data by hand. For example, use identifiers instead of full strings. For example in a script or module, assign individual items to numerical identifiers instead of just saving the full item name. Likewise if you’re using things like properties assigned to objects (like a part’s properties like colours, materials…), write out a table of identifiers that correspond to a string.

local props = {
    'ClassName';
    'Material';
    'Color';
}

-- to serialize:
local data = {}
for i,v in ipairs(workspace:GetDescendants()) do
    local thisPart = {}
    for i,property in ipairs(props) do
        thisPart[i] = v[property]
    end
    table.insert(data, thisPart)
end

-- to deserialize,
for i,v in ipairs(data) do 
    local newInstance = Instance.new(v[1])
    for index, propertyName in ipairs(props) do
        if index == 1 then continue end
        newInstance[propertyName] = v[index]
    end
end

After that, you can look into data compression algorithms, you want a lossless data compression method. There are so many data compression algorithms that exist but some that I can think of off of the top of my head are LZW, ZIP (yup like the .zip file format), and gzip but there are many, many more. And as I have mentioned, some guides and full resources exist on this forum:

Most efficiently saving data

I’m familiar with using DataStores and saving data, but I’m interested in one thing: I’m going to be temporarily saving a player’s progress in my tycoon. I could save the name of each thing that they have bought: Is saving a table such as {tick(), “Dropper1”, “Dropper2”, “Dropper3”} which could have upwards of 100-200 items in the list a horrible way to achieve this? How could I make this more efficient? Perhaps milestones and then save some kind of integer as to how far they’ve gotten? Ther…

Compression for limited data (Datastore)

Are you sure that that other way will not take up too much space then? Anyways, you can either listen to my (probably bad) explanation Or watch the video I linked before, if you haven’t done that already. Basically, huffman is a way to compress characters (or other “single” token stuff, like block IDs) into a long binary string. First you have to construct a huffman tree, I’ll explain that a bit later on. It will look something like this: [image] How to decode the binary string: Read fro…

Text compression

I wrote a text compression program which can be used to fit more data inside of a datastore key. It’s based on the LZW algorithm and uses the 127 different characters from \1 to \127 that datastores support. edit: Some characters take up more than one character worth of space when encoded and stored in a datastore. Control characters mostly take up 6 characters each, and " and \ each take up 2 characters. This has been updated to avoid using such characters in the compressed output. 93 unique c…

Do bear in mind that you have 4KB of space per key, that is a lot of data, 4 million characters. Depending on what you’re trying to compress, different algorithms will be more effective than others, and some will also not work at all and will actually increase the size of the string, however all in all you will find data compression will be more useful if you’re saving a lot of the same thing which is how identifiers come into play. If you do reach the limit though, you can also always break up your data between keys.