Datastore limits

Help and Feedback Scripting Support
#1

Hey, I’m developing a very large scale game on roblox (MMO RPG) as a passion project, but however I have come across the need rather large save file. The size of the save could be upto 10 MB in some cases since there will be a lot of customisation.

Roblox Datastores allow up to 260 KB of data to be stored in one key, and this has a access limit, to save and load would take 40 gets / sets.

My other consideration is that you have have 500 http requests per minute with a max post size of 1 MB each, so this would mean I’d have to split it into 10 separate save files and repack them, this would mean using my own external servers and scaling them as well as doing maintenance. This will also cause latency when teleporting.

Any suggestions? or is external servers the way to go.

4 Likes
Help saving user-modified terrain
#2

Why do you need 10 MB to store player data?

That’s quite a lot - are you storing the least amount of information possible? Are you storing, for example, an entire weapon (its stats, textures, etc.) in the DataStore?

For something like the latter, you should just store its ‘ID’ with maybe a little bit of customisation info, then retrieve the persistent info like stats from a module in ServerStorage or something using its ID.

6 Likes
#3

You can always compress the data to save up on space.

3 Likes
#4

For inventory it’ll be ItemID,Amount,AmountStolen
but there are tons of features like customisation and lists of lose items left lying around your house like in skyrim, I’d have to record ItemID,Orientation,Poisition,MapLocation.

As well as all perk trees, stats and more. I will use simple datastores until I need to scale above 260 KB, but if you look at skyrim’s save files it ads up to around 26 MB of data, since we are recording the same sorts of things that’s what it could be. I will be compressing as much as I can though and despawning loot in chests after a certain amount of ingame time.

2 Likes
#5

Hia. As someone who is constantly dancing on the edge of reasonable use of datastores, I’m here today to tell you that there are no data limits for DataStores!!

The only limit is a individual value character limit. You can have no more than 260,000 characters in a single datastore value. Your data is automatically converted into a JSON string when you save to datastores and you can use HttpService.JSONEncode to see how many characters your files would have.

23 Likes
Would an array be more size-friendly than a dictionary?
#6

Yes but splitting it into 40 datastore values would use up a lot of the allowance for GetAsync and SetAsync, both at 60 + (10 * players) per min respectively.

#7

If you’re using 260,000 for a single save file… I’m sorry but you are doing something wrong.

The most I’ve ever used for my game is around 200,000 when someone placed over several thousand potted plants on their base and I had to save the position and rotation of each plant.

This was before I learned about compressing floats from something like 2.4999999999999999984 to 2.5. If you are using any floats (non-integer numbers) in your data files you should compress them with something like this:

-- converts a long and messy float into a neat and short STRING
-- there's probably a better way to do this though

local function formatfloat(float)
	local str = tostring(float)
	local pos = str:find("%.")
	if pos then
		local pre = str:sub(1,pos-1)
		local suf = str:sub(pos+1, pos+2)
		if pre:len() > 0 and suf:len() > 0 then
			local sufval = tonumber(suf)
			local preval = tonumber(pre)
			if sufval and preval then
				local inc = 1
				if str:sub(1,1) == "-" then
					inc = -1
				end
				suf = "0"
				if (sufval >= 40 and sufval <= 60) or (sufval >= 4 and sufval <= 6) then
					suf = "5"
				elseif sufval >= 9 or sufval >= 90 then
					preval = preval + inc
				end
				return (tostring(preval) .. "." .. suf)
			end
		end
	end
	return float
end

edit: this function rounds to the nearest .5

18 Likes
#8

For the average item it will take this much data [999,99,99] instead of [“ItemID”:999,“Amount”:99,“Stolen”:99] so each inventory record would take approx 11 bytes, 12 if you include the “,”. And so say you had 1000 items in your inventory, that’s 12kb, and you had 200 chests recently looted item 50 items remaining in each, that’s another 120kb (200 x 50 x 12b). now we have character customisation, 1kb max ish, skills, perks, stats total around 2kb.

Now I’m at 135 KB.

Now for quests, format being [QuestID,QuestPos] or [999,99] or 8 + 1 for comma = 9 B per quest,
after a long play time you may have done 200 quests so that’ll be 1.8 KB = 136.8 KB

Now for player houses, let’s say the base house details take 1.2 KB for the main part of, = 138 KB
Now I have to account for each lose item, [ID,Amount,X,Y,Z,LocationID] or [999,99,999.9,999.9,9,999] = 26 B + 1 for comma = 27 B, some players with a ton of stuff lying around (200) = 5.4 KB = 142.2 KB

That’s less than I thought, but that’s just for the game base, later on there will be tons more which could push it into the 1 Mb - 3 Mb area

EDIT: 1 - 3 mb is actually not that bad, as that’s not many shards

1 Like
#9

You could also use string.format for this. e.g
print(string.format("%.4f", math.pi))

9 Likes
#10

There is absolutely no reason for you to be saving key names for every single entry to the table. You already know what each entry is and you are just wasting space.

[“ItemID”:999,“Amount”:99,“Stolen”:99] -> [999,99,99] already saves you a massive amount of space.

Edit: There’s also no reason for you to be recording information about items the player doesn’t have. Save space by not putting in a table record for any items where the amount and stolen are 0

2 Likes
#11

Yes, that’s what I said, that I was already compressing it down to [999,99,99] as an array instead of a dictionary

Edit: I’m not recording anything they don’t have, it’s an array of [id,amount,stolen],[id,amount,stolen],[id,amount,stolen] etc

#12

Oh gotcha, my bad. I think if you limit the amount of items someone can have in their inventory (reasonable for an mmo to do) you shouldn’t come anywhere near hitting the character limit. If you do, you can just split the base-building stuff into its own datastore.

1 Like
#13

I’m actually adopting your ordereddatastore method for saving to make sure it works properly too, as losing 100 hours of playtime data with items you paid for would he a nightmare for players

1 Like
#14

Another trick is to get rid of redundant item ids by nesting item attributes tables in a single table per item. My keys look like this:

  • Item 1
    • {Amount0, Stolen0}
    • {Amount1, Stolen1}
    • {Amount2, Stolen2}
  • Item 2
    • {Amount0, Stolen0}

etc.

But yeah I’m also questioning what kinds of stuff you’re trying to save that you need 40 keys for.

1 Like
#15

I have to save player data over multiple keys in Farming among Friends but I think only a few users ever had so much data that it was really necessary. I never thought about floats being saved as crazy decimals so I’m sure that hurt the size of the data. I also just stored plain JSON instead of some compressed format and that also doesn’t help.

#16

@berezaa @TheGamer101
This is unrelated to the OP but since you guys posted here, I’ll throw it out there. Can’t you just use math.abs(num) to get rid of floating point errors? I did that for the values of NumberValue objects.

1 Like
#17

It appears so.
image

4 Likes
Players data store doesnt save
#18

Appears not:

local http = game:GetService("HttpService")

local str = http:JSONEncode({0.1+0.2})
print(str)
-- [0.300000000000000044408920985006]

local str = http:JSONEncode({math.abs(0.1+0.2)})
print(str)
-- [0.300000000000000044408920985006]
1 Like
#19

you can convert to string and truncate after 1 decimal place

#20

I know, but I was directly referring to the proposed solution of math.abs… which doesn’t work.