Most efficiently saving data

Help and Feedback Scripting Support
#1

I’m familiar with using DataStores and saving data, but I’m interested in one thing:

I’m going to be temporarily saving a player’s progress in my tycoon. I could save the name of each thing that they have bought:

Is saving a table such as {tick(), “Dropper1”, “Dropper2”, “Dropper3”} which could have upwards of 100-200 items in the list a horrible way to achieve this? How could I make this more efficient?
Perhaps milestones and then save some kind of integer as to how far they’ve gotten?
There aren’t any duplicate items

In general, what are the best practices when it comes to saving data? Should I be saving keys (and then looking them up in a dictionary when retrieved) to keep minimal amount of data in a datastore? Thanks.

2 Likes
Data efficiency
#2

Each key can store up to 250,000 characters, so you can save a few hundred items in one table without it being an issue. However, I would recommend assigning IDs to each item and storing that ID. Then using the IDs to load each of the items that the player owns.

3 Likes
#3

200 items why? What are the specifics you’re saving in a DataStore? DataStores might have a 260K+ character limit (cc @Spynaz, you’re off by 10k), but it seems to me that you’re trying to cram all data a player has into a single key rather than spreading it out across different keys and scopes.

3 Likes
#4

You could do soemthing like this but I really see no real need (this is just code I put together). This would need a fixed array of droppers.

local data = {1,1,1,1, 0,0,0,0, 1,0,1,0, 0,0,1,1, 1,1,1,1, 1,1,1,1}
print(unpack(data))

local function save(a,b,c,d,e,f,g,h)
	return a + 2 * b + 4 * c + 8 * d + 16 * e + 32 * f + 64 * g + 128 * h
end

local function load(val)
	local b,c,d,e,f,g,h = 0,0,0,0,0,0,0,0
	if val > 127 then h = 1 val = val - 128 end
	if val > 63 then g = 1 val = val - 64 end
	if val > 31 then f = 1 val = val - 32 end
	if val > 15 then e = 1 val = val - 16 end
	if val > 7 then d = 1 val = val - 8 end
	if val > 3 then c = 1 val = val - 4 end
	if val > 1 then b = 1 val = val - 2 end
	return val,b,c,d,e,f,g,h
end

local tbl = {}
for i=1, #data, 8 do
	tbl[#tbl+1] = save(data[i], data[i+1], data[i+2], data[i+3], data[i+4], data[i+5], data[i+6], data[i+7])
end

print(unpack(tbl))

for i=1, #tbl do
	print(load(tbl[i]))
end

Hope this helps

3 Likes
#5

I was working on nearly the same thing when you posted this code. Here is my untested version:

local function toBinary(bits)
	local bytes = {}
	local maxByte = 0

	for i in next, bits do
		local bit = i % 8
		local byte = (i - bit) / 8
		if byte > maxByte then
			maxByte = byte
		end

		local value = bytes[byte] or 0
		byte[byte] = value + 2^bit
	end

	for i = 1, maxByte do
		bytes[i] = bytes[i] and string.char(bytes[i]) or "\0"
	end

	return table.concat(bytes, 1, maxByte)
end

local function toBits(binary)
	local chars = {binary:byte(1, binary:len())}
	local bits = {}

	local curPos = 0
	for byte = 1, #chars do
		curPos = 8 * byte
		local value = values[byte]
		while value > 0 do
			local bit = value % 2
			if bit == 1 then
				bits[curPos] = true
			end
			curPos = curPos + 1
			value = (value - bit) / 2
		end
	end

	return bits
end

-- example:
-- starts at 0, sparse array of truth values
local data = toBinary {[0] = false, true, false, true, true, nil, [200] = true}

-- returns nearly the same table, but with all falses replaced by nils
local sameData = toBits(data)

A compression algorithm should be able to compress this data to make it very, very small since it’ll probably contain lots of zeros. Note: this may not actually work if Lua/datastores don’t handle embedded null characters well. Lua bytecode has an explicit string length stored for every string, but idk how the other programs down the pipeline handle them. You could use numbers instead, but those have a fixed length and you can only store one number per a field.

2 Likes
#6

I’m just saving every item the player has bought, a long with the time it saved, in an attempt to temporarily save their data if their game crashes/they accidentally leave. Surely spreading the data across multiple keys would just result in more calls of GetAsync?

#7

I’m not too familiar with compression. Care to explain more what’s happening?

I can tell that you are compressing each string into a byte in order to save less data.

#8

JSONEncoding and then compressing it should be more than enough.

My suggestion is that you give unique id to each dropper type and save using that id, to lower the char count.
If you have something like “Supercalifragilisticexpialidocious Dropper”, it’s definitely better to store it as “A01” than it’s name.

4 Likes
#9

Think of this as a bit shift operation we are converting base 10 - > base 2. 1,0,1,0,1,0,1,0 = 170 we shift each value to the end of our 1 byte(8 bit) value the first shift would be 1,0,0,0,0,0,0 = 1 << 7 = 128 the next would be 1,0,0,0,0,0,0 = 0 << 6 = 128 then 1,0,1,0,0,0,0 = 160 ect until we reach the end value of 170 (base 10).

We can inverst the step to get the value at each point 170 >> 7 = 1 so we take 128 from the value current value and repeat until we reach our bit value (base 2).

1 Like
#10

I wouldn’t say your method is horrible, but it can definitely be improved to use less memory.

As you use predefined Droppers, it is possible you can assign a number value or a string to a dropper, i.e:

{tick(),1}

in a module script elsewhere on the game…

{ [1] == “Dropper1”; [2] == “Dropper2”}`

Also, it can’t hurt to do a bit more management with your table. Maybe assign another value to that tick instead of having it in the open, and put all of the other points in a separate array:

{
tickValue = tick();
ownedDroppers = {1,2,3};
name = "I read you might want to include a name so do it here";
}

It really depends on how you want to store it. This would be ideal for a game that might include multiple saves and as a result you need to include separate data entries for each character. However, it also works for your game too!

2 Likes
Can You Save Buildings With DataStores?