I’m currently trying to find the best way to compress huge tables which can contain more than over 1 million characters, and save them to my datastore. Is this possible? What’s the best way of doing this?
Try serializing the data
1 Like
How would I apply this to my case? My tables consist of a LOT of part names and cframes.
This is what they look like
Why not make table a string?
local http = game:GetService("HttpService")
local tabl_e = {["aofwoiajfoiwajfow"] = 1234, ["test"] = "hi", ["users"] = 2140}
local table_as_string = http:JSONEncode(tabl_e) -- makes the table Json string
-- Getting the table back
local decodedTable = http:JSONDecode(table_as_string)
print(decodedTable.test) -- prints hi
1 Like
Does making a table into a string reduce memory usage enough for my table though? This is actually my current method, I’m just looking if there’s any way to pack it down more.
Yeah, it does.
Thats how serialization works.
Every character in a string takes 8 bits
while an empty table takes 64 bytes and each item in the table takes 16 bytes of memory.
You can learn more about it here.
Any how, people use serialization to store big tables and Vectors,CFrames,Color3 Values etc.
And it works fine and wont take much space since its a single string.
2 Likes
So, how would this perform being saved through datastore?
First of all, this data is for keeping track of a players character position. This is only data from 8 seconds of tracking.
For what I found, the maximum amount of characters is 4 million. This seems like a lot compared to my use, but If I’m recording players movement for up to 60 seconds + I’m scared that’ll surpass the limit…
You can split your data, cause 1M+ is alot of data.
Well one thing I can think of is instead of using cframe, I could just get rotation and position and leave out the other parts. Instead of keeping the name of the players characters part that’s being moved, I could just give it a number which can be found in a table as the parts name.
Is there anything else?
Its still gonna be alot of data.
You do have to split it.
I could have it so it updates less, and have the client tween between movements to make up for it
You can try, is data key is same for every player or its different?
Each key is different.
If you’re wondering what I’m using this for, I’m trying to make it so you can “record”, which just keeps track of the positions of workspace (currently only the players character), and playing it back. Parts other than the character will be added after because I’m looking to make them update a lot less.
Oh, I see.
Well then you can try make sure not to add much values to the data as possible.
Btw you can use ProfileService its a great module it have datastore2 too. It might help alot.
I marked your response a solution above, I’m just going to try to make all the names of parts be numbers, and update a lot less and have tweening make up for it. Thank you!
Edit: I’m really sorry to bump this, but I’ve significantly compressed my tables.
Before, a 6 second recording in a table took up more than 1 million characters. After a lot of research, I’ve managed to make a 60 second recording take up only 344k characters. That is a HUGE improvement!
(About 29 times better)
Here’s a couple things I’ve done:
Instead of using names, I keep a table like this since the parts will always be the same, and I can convert names to numbers and back
For my Cframes, I really only need the rotation and position so I insert position and rotation into a table, and can convert them back later.
function Compressor.LightCFrameCompress(pos,rot)
return {pos.X,pos.Y,pos.Z,rot.X,rot.Y,rot.Z}
end
Now, I could just use HTTP service to JSONEncode the table, but I found a way to compress the compression.
Using a text compression resource, I can compress the JSONEncoded table (I made his script work with a module script)
Here’s what it looks like while completely compressed:
I can Decode this simply using
And bam!
To decode my cframes, I can use this
function Compressor.LightCFrameDescompress(tbl)
return CFrame.new(Vector3.new(tbl[1],tbl[2],tbl[3]),Vector3.new(tbl[4],tbl[5],tbl[6]))
end
Don’t take my methods for granted as I am not sure this is the best way to do this, but it works perfectly for me!
4 Likes
kind of late, but you shouldnt store cframes as a whole in a datastore. If you really do need it then try rounding up some of the numbers things like 0.00000001 can take a lot of unnecessary space.
Another thing for compression that will heavily lower your space is use a base 90 numbering system.
Example:
5 character string can hold that amount of data which is pretty nice
Major downside is that it only works for integers
I suggest this video about data compression
