How to compress huge tables? (more than 1M characters)

I’m currently trying to find the best way to compress huge tables which can contain more than over 1 million characters, and save them to my datastore. Is this possible? What’s the best way of doing this?

Try serializing the data

1 Like

How would I apply this to my case? My tables consist of a LOT of part names and cframes.

This is what they look like

(There can be far past 1000 indexes)

Why not make table a string?

local http = game:GetService("HttpService")
local tabl_e = {["aofwoiajfoiwajfow"] = 1234, ["test"] = "hi", ["users"] = 2140}

local table_as_string = http:JSONEncode(tabl_e) -- makes the table Json string
-- Getting the table back

local decodedTable = http:JSONDecode(table_as_string)
print(decodedTable.test) -- prints hi
1 Like

Does making a table into a string reduce memory usage enough for my table though? This is actually my current method, I’m just looking if there’s any way to pack it down more.

Yeah, it does.
Thats how serialization works.

Every character in a string takes 8 bits
while an empty table takes 64 bytes and each item in the table takes 16 bytes of memory.

You can learn more about it here.
Any how, people use serialization to store big tables and Vectors,CFrames,Color3 Values etc.
And it works fine and wont take much space since its a single string.


So, how would this perform being saved through datastore?

First of all, this data is for keeping track of a players character position. This is only data from 8 seconds of tracking.

For what I found, the maximum amount of characters is 4 million. This seems like a lot compared to my use, but If I’m recording players movement for up to 60 seconds + I’m scared that’ll surpass the limit…

You can split your data, cause 1M+ is alot of data.

Well one thing I can think of is instead of using cframe, I could just get rotation and position and leave out the other parts. Instead of keeping the name of the players characters part that’s being moved, I could just give it a number which can be found in a table as the parts name.

Is there anything else?

Its still gonna be alot of data.
You do have to split it.

I could have it so it updates less, and have the client tween between movements to make up for it

You can try, is data key is same for every player or its different?

Each key is different.

If you’re wondering what I’m using this for, I’m trying to make it so you can “record”, which just keeps track of the positions of workspace (currently only the players character), and playing it back. Parts other than the character will be added after because I’m looking to make them update a lot less.

Oh, I see.
Well then you can try make sure not to add much values to the data as possible.

Btw you can use ProfileService its a great module it have datastore2 too. It might help alot.

I marked your response a solution above, I’m just going to try to make all the names of parts be numbers, and update a lot less and have tweening make up for it. Thank you!

Edit: I’m really sorry to bump this, but I’ve significantly compressed my tables.

Before, a 6 second recording in a table took up more than 1 million characters. After a lot of research, I’ve managed to make a 60 second recording take up only 344k characters. That is a HUGE improvement!
(About 29 times better)

Here’s a couple things I’ve done:

Instead of using names, I keep a table like this since the parts will always be the same, and I can convert names to numbers and back


For my Cframes, I really only need the rotation and position so I insert position and rotation into a table, and can convert them back later.

function Compressor.LightCFrameCompress(pos,rot)
return {pos.X,pos.Y,pos.Z,rot.X,rot.Y,rot.Z}

Now, I could just use HTTP service to JSONEncode the table, but I found a way to compress the compression.

Using a text compression resource, I can compress the JSONEncoded table (I made his script work with a module script)


Here’s what it looks like while completely compressed:

I can Decode this simply using

And bam!

To decode my cframes, I can use this

function Compressor.LightCFrameDescompress(tbl)

Don’t take my methods for granted as I am not sure this is the best way to do this, but it works perfectly for me!