Create an Automatic DataStore Compression Layer

As a Roblox developer, it is currently too hard to live with the current DataStore size limits (even with the current size increase to 4,000,000 chars).
I have seen several topics discussing ways to compress strings using various algorithms.
The problem is that JSON does not allow the storage of all allowed values ​​in one byte (0 to 255); thus, a compression system is limited, even using Base64, it generates more final characters than the original ones, and instead of compressing, Base64 expands the original size. Even using LZW compression algorithms, if there is no sequence repetition, nothing will be compressed.

This whole problem could easily be solved if Roblox placed an automatic (and invisible) compression system, a layer between the DataStore server and the game. That is, the programmer would no longer have to worry about manual compression algorithms and all data sent by the game to be saved (JSON) would pass through this automatic compression layer and would be saved in binary, compressed format. And in a reverse way, when the game reads the data, the compression layer will automatically decompress the saved bytes and return the pure JSON string.

If Roblox is able to address this issue, it would improve my development experience because, in this format, everyone wins: Programmers no longer have to worry so much about manual data compression (except if the programmer wants to insert a manual data dictionary system) and at the same time Roblox will have much less space being occupied on their servers.

9 Likes