Hello, I plan to use Roblox Datastores to save information that isn’t linked to a specific Roblox player(like how much currency they have) but the entire player base(like for example houses they have placed on the map) and thus because I want the data to load quickly I have to save as much data I can in a single datastore key so I can load as many information as possible at once. I’m wondering if Roblox compresses information that it saves in their datastores or not and how and also if there’re modules out there that can compress and uncompress dictionaries using known methods(so I don’t have to reinvent the wheel). From calculations alone I have determined that I can fit the data of 15k players in one datastore key, however, I need to optimize that as much as possible, because if for example, 100k different players place a house, I’ll have to make 100//15+1 = 7 datastore calls to load the information with the current uncompressed/unoptimized method of saving the data.
Roblox doesn’t compress data, or if it does, it doesn’t affect the DataStore limitations. DataStores are saved as raw JSON and your data size can be calculated by print(string.len(encodedJSON)). Each character is one byte, so with the limitation for 4mb, you have 4 million characters worth of space per player DataStore. Things like brackets, commas, key names, all of that takes up space as well. You’ll have to use your own method of compression.
Also the amount of requests made are 60 + (number of players * 10) per minute, for read and write each.
Due to the above link no longer being valid, here’s a better source with updated links:
Does number of players refer to the max players of an experience or the current amount of players in the server?
The current number of players in the server.
Are there community libraries that compress and decompress dictionary information?
Also if there is no compression, does that mean that Base64Encode and Decode is one method of compressing the data to an extent?
I’m not sure of that myself, but I’m sure you could find something if you search on the forums and perhaps elsewhere. But really, the main thing to consider is the order of information. That way, you can just save your data as a long string with split identifiers. You can also do things like number compression, if you want to compress large numbers into bytes (basically using letters that represent numbers).
An example of using both at once, you can look at a Minecraft chunk as an example. A chunk is 16x16 wide and 256 tall. To save a block, you’d only need about 4 bytes. A byte is 8 bits, where a bit is the 0’s and 1’s. A byte can have 256 different combinations of bits. You can use 2 ^ bits to determine the number of possible combinations. So to store the position, you’d just use the block’s relative position within the chunk. So the Y axis is one byte. The X and Z axis are 16 each, and half a byte (2^4) has 16 possible combinations, so you can use one byte for the X and Z axis. Then the rest of the bytes would determine the block id and perhaps other things like block state. For example, 13 bits for the id for 8k possible block IDs and 3 for the block state (4 is probably enough but you never know), which would only cost 2 bytes.
Using everything above, that changes BlockData={X=16,Y=256,Z=16,Id=8192,State=8} to just a string of 4 characters (4 bytes). Instead of putting commas, (during load) you’d just split the string up into groups of 4 characters, because you know 4 characters is going to be a block. So you won’t have to worry about commas or anything else.
If you are handling numbers in your data, you can indeed compress it by encoding the numbers into a higher base when saving and then decoding it back to decimal when loading.
Since the length of the data when JSONEncode is used determines how much you’ve used, you can try to remove unnecessary characters to save space. An example could be a value that is rounded if you don’t need the value to be too precise. If you save table values for something in your game that gets saved in order like a part with Name, Color, Transparency, you can remove extra commas and brackets since you know to get the data of a part by grouping every three values in the table.