Lower Database Storage by 400%

Resources Community Tutorials
#1

Hello all, :happy4:

This is a tutorial on how to compress your datastores by as much as 400% using the LZ4 Library and Buffer. :wink:

Prerequisites
LZ4 Library

Step 1: Have a large amount of text you want to compress.

image
image1115×562 80.8 KB

This string is about 35000 characters long.

Step 2: Using the LZ4 library, compress the string using .compress() . The output should look something like this.

image
image797×502 84.7 KB 

local LZ4 = require(script.LZ4)
local LZ4compressed = LZ4.compress(String)

This string is only about 8700 characters long.

Step 3: Compress the string again using buffer.fromstring() and you should get something like this.

buffer: 0xae543138174fb25f

local LZ4 = require(script.LZ4)
local LZ4compressed = LZ4.compress(String)

print(buffer.fromstring(LZ4compressed))

Step 4: Save this buffer into the database you want and to should do something like this

local yourDataStore = game:GetService("DataStoreService"):GetDataStore("String")

local LZ4 = require(script.LZ4)
local LZ4compressed = LZ4.compress(String)

yourDataStore:SetAsync("String",buffer.fromstring(LZ4compressed))

local buffercompressedData = yourDataStore:GetAsync("String")
local LZ4compressedData = buffer.tostring(buffercompressedData)

print(LZ4.decompress(LZ4compressedData))

We can save tables in LZ4 using game.HttpService:JSONEncode, here’s an example.

local yourDataStore = game:GetService("DataStoreService"):GetDataStore("String")

local Table = {
	["Lorem"] = {"ipsum"},
	sum = 124124,
	{} 
}
local LZ4 = require(script.LZ4)
local LZ4compressed = LZ4.compress(game.HttpService:JSONEncode(Table))

yourDataStore:SetAsync("String",buffer.fromstring(LZ4compressed))

local buffercompressedData = yourDataStore:GetAsync("String")
local LZ4compressedData = buffer.tostring(buffercompressedData)

print(game.HttpService:JSONDecode(LZ4.decompress(LZ4compressedData))) -- {["Lorem"]={"ipsum"},sum = 124124,{}}

Credits to @metatablecatmaid for the port of LZ4!

LZ4 Download Link:
lz4.lua (7.0 KB)

12 Likes
#3

For the small storages i think it will be worse, cause the lz4 will benefit from the original size. I you are going to encode only small values it will only increase the data size i guess

#4

as well if you doing the compression i find the zlib very good for these cases

#5

i dont know if its safe to use this because arent some of those characters non-utf8? i also thought buffers shouldnt be used for datastores for the same reason…

i ran the code

local yourDataStore = game:GetService("DataStoreService"):GetDataStore("String")

local Table = {
	["Lorem"] = {"ipsum"},
	sum = 124124,
	{} 
}
local LZ4 = require(script.LZ4)
local LZ4compressed = LZ4.compress(game.HttpService:JSONEncode(Table))

yourDataStore:SetAsync("String",buffer.fromstring(LZ4compressed))

local buffercompressedData = yourDataStore:GetAsync("String")
local LZ4compressedData = buffer.tostring(buffercompressedData)

print(game.HttpService:JSONDecode(LZ4.decompress(LZ4compressedData)))

and it returned this

image

#6

I only used buffers for storage because of non-utf8 since trying to store non-utf8 in a database returns an error plus an added bonus of extra compression.

#7

This is cool, but i dont know how well it will work when you want to view the data itself you’ll probably have to convert what you want?

#8

Yes you’ll have to convert it back.

#9

using base 64 is always an option :wink:

#10

but base64 would just end up increasing the storage even more, as every 3 bytes of binary data are encoded into 4 bytes of base64.

but yes, easier to implement, but you’d definitely see an increase in storage.

#11

best option in my opinion is serialization and hexadecimal conversion.

for example, a player has an inventory with 100 gold pickaxes and 200 diamond ones.

you can represent this as simple as *1:100*2:200. * is the identifier between items, the following number up to the colon is the ID, and the number after the colon is the amount.

you can even go further and turn your IDs and quantities into hexadecimal which would look like *1:64*2:C8.

(id of 1 is for gold pickaxe, 2 is for diamond)

there is another step for people who need even more compression, but i’ve never found it necessary. roblox gives you >4 million characters to store data in, and with my previous example you just stored 300 generic items in only 10. plus you have the added bonus of being able to read the serialized data and understand it without having to run it through a decryptor.

if anyone has other steps id love to hear them.

#12

I would do this for a personal datastore, but im just wasting Roblox’s storage doesn’t really affect me lol.

2 Likes