Questions About Data Saving

Help and Feedback Scripting Support
1

Hello so, as you may or may not know, I have made a Roblox Game where you could make Animations on Lower End Devices, such as, Mobile Phones, more information on that is on this topic

I would like to Implement Animation Saving for it, for convenience.. But I have never touched DataStore ever in my life, but based from my research, there’s a limit of 1 Million Bytes or, 1 MB (correct me if I’m wrong), which is definitely.. not enough, even for the 800 Keyframes limit, as one Keyframe may contain the data of multiple objects, unless i can figure out a god tier serializer, this doesn’t seem to be a good choice

I have thought of an Idea that may sound.. stupid, but, might actually work,

Basically, when players tries to “Save” their animations, it will put in a textbot the Serialized/Compressed Animation Data, as a String, for which the Player can Copy and Store to a website similar to Pastebin

And then to Load the animation, they could Input the Link which contains Raw Text of the Data, or the Entire Data String, In which a script will deserialize it and turn it into what it needs to be, and no, I will make sure this won’t be used for exploiting

As stupid as it may sound for you DataStore experts, is this a viable option? Or is there a better alternative? Any questions or confusions, tell me and I’ll try my best to answer

1 Like
2

You could make 800 keyframes weight 184KB (with 10 parts) if you can serialize objects into an 8bit id and guarantee that all names of the animatable parts will be unique unless I messed up the calculation, but even then that’s too much for DataStores and you’d need to put some extra data before so it would weight more than that

2 Likes
3

The website says 4MB per key. Worst case you can shard your keyframes over multiple keys because there’s nothing stopping you from doing that. Although Roblox recently rolled out a total datastore limit, you will not have to worry about it unless you’re saving obscenely large amounts of unoptimized data.

1 Like
4

I don’t really think that you could reach the 4MB even with 800 keyframes unless the keyframes hold like a LOT of data and even if it is possible i don’t think most people would even ever reach that.
However if 4MB isn’t enough try sharding the animation over multiple keys as @pullman45 said

but if you’re going with like a ‘save it to pastebin’ method i suggest just serializing the data and giving it to the user then deserialize it when they want to open the animation again

1 Like
5

i think roblox datastore saves with json, so compressing into your own string can save data

you can convert cframe components into integers which can be compressed into strings with formats like ascii85 using this module i found (dont remember credit/source)

local CustomNumbers = {}
CustomNumbers.__index = CustomNumbers

function CustomNumbers.new(alphabet)
	local Numbers = {}
	setmetatable(Numbers, CustomNumbers)
	Numbers.Alphabet = alphabet
	Numbers.AlphabetLength = #alphabet
	
	return Numbers
end

function CustomNumbers:Encode(x)
	local result = ""
	if x < 1 then return result end

	local quotient = x
	local remainder = -1
	
	while quotient ~= 0 do
		local dec = quotient - 1
		remainder = dec % self.AlphabetLength
		quotient = math.floor(dec / self.AlphabetLength)
		result = self.Alphabet[remainder + 1] .. result -- Compensated for lua's index 1 start
	end
	return result
end

function CustomNumbers:Decode(x)
	local sum = 0
	local index = 0

	for i = #x, 1, -1 do
		local c = x:sub(i,i)
		sum += table.find(self.Alphabet, c)*(self.AlphabetLength^index)
		index += 1
	end

	return sum
end

return CustomNumbers

edit: this is the source CustomNumbers - Convert your base 10 numbers to a custom alphabet

1 Like
6

I think you can pretty easily reach the 4MB limit, right now each Object Data looks like this

C = CFrameHere
S = SizeHere

(Of course, I’m working on a better compressor/serializer, there’s none of that YET)

And, that is each object.. one keyframe can easily have 30+ Objects, so times that by the limit of 800.. It will probably not reach the limit, but be SCARILY close

7

Yea I’ve thought of that, but didn’t found a source code to work with, so thanks!

8

Yeah, I’ve thought about that, I’ll give it a shot if i REALLY have to, once i start making the DataStore system.. right now, still on the planning process

9

If we say that a CFrame can be stored in 28 bytes (based off sera’s lossy CFrame), and the size as 24 bytes (three 8 byte floats), that would be 52 bytes per object per keyframe. So for 800 keyframes, and 30 objects, that would be 1 248 000 bytes

You could store the size as three 4 byte floats, (12 bytes per vector), and that would lead to 960 000 bytes

Without the size, it would be 672 000 bytes (genuine question, why do you have to store the size for animations? Does the animation include the size of objects?)

(I am not converting the amount of bytes into MB or MiB, because it’s a mess whether 1kb is 1024 bytes, or 1000 bytes. I think the limit of 4 MB is the former, aka 4 194 304 bytes. There is different notation to distinguish the two (mb MB or MiB), but I forgot which one is which, and respect of this notation was thrown out the window)

This is a lot of data, but it’s still within the limits, if you give each animation it’s own key. Especially if the 800 keyframes with 30 parts example is the upper bound of what you’d expect, it’ll probably be fine. I doubt you’d run into the throughput limits (25MB read per key per minute, 4MB write per key per minute) if you cache your results, or the storage limits (it’s very generous). The GetAsync() limit could be more problematic if you need to load tons of animations (60+ in less than a minute)

Datastore limits

Given the structure of the data you are saving, my guess is that from Keyframe to Keyframe, many values wont have changed, so you could try to store only when the value changes. You could also use a compression algorithm such as this luau lzw text compression, which are most effective when the data has lots of repeating patterns

1 Like
10

Pretty much.. you can make the sizes of objects transform during the animation, if you want to

Yeah, I already thought a lot of things through when i wanted to optimize it to the max.. Including this, each keyframe only store the values of objects if it’s moved in that keyframe… If it makes sense.. basically just what you mentioned there

I did also thought about limiting how many objects you can move per keyframe, but :grimacing:, I don’t know how i feel about it, it’s either limit how many objects you can move per keyframe, or limit how many keyframes you can make even more, I have yet to find the optimal and most reasonable numbers/limit for both cases

1 Like
11

By using HttpService:JsonEncode(), and getting the length of the string it returns, you can check if the data exceeds or not the datastore size limit (for buffers you can do buffer.tostring() and JsonEncode that I think). With this, you can implement a data limit instead of a limit on keyframes or objects, and you can even show the player how much data their animation is taking up, as a %

1 Like
12

That’s actually a… pretty good idea! Thanks! I’ll give this a shot^^

1 Like
13

Also, thanks to everyone here, I’ll make sure to use all the Ideas if it fits the project!!

1 Like
14

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.