How do big games send large amounts of data store requests without clogging the queue up?

Help and Feedback Scripting Support
#1

How do big games like Pet Simulator X! load large amounts of data from data stores so efficiently?

I am trying to achieve a Purchase History system for the Traveliing Merchant in my game Kastle Kingdom!. I already have all the scripts working as intended, I’m not asking for a purchase history saving system, I’m asking if there’s a better way to handle large amounts of data.

So my game has a lot of data value.
“Last Position”
“XP”
“Health”
“Hunger”
(all the purchase data with different keys according to the purchase number (the order it was purchased in).
IsBanned
BanReason
TimeBanned (minutes)
HasJoined
SHasJoined
ShopHasJoined
And many more.

I did make the purchase data wait like 5 seconds before loading so all of the important stuff can be loaded first.

But when it loads it, it can load about 10 or so with the random wait time of around 0.1-0.5, and then it starts throwing that it has been added to the Data Store queue.

How do big games like Pet Simulator X! load and save so much data at once without clogging up the data store queue?

To my knowledge, Pet Simulator X saves and loads the pet inventory, the pet index (what pets you have and what ones you don’t), the mailbox history, Trading history (pets / gems traded, usernames in trade) How many days ago it was traded, and the Usernamee of the person you traded with, wether you liked the trade or not, the mailbox history (pets / gems traded and username), Mailbox items (custom message player left) (amount in gems / pets) username, the bank history and everything in the bank, and way more. Just so much, how can a big game handle so much data store requests efficiently while my game can only handle like 10 from the purchase history and like 30 from the data above?

1 Like
#2

put your data into one single table so you only need to call save once. you can use JSONEncode (JSONDecode when loading) to size down your data a bit

5 Likes
#3

How would I go about doing that? I’ve never really saved data stores in tables.

2 Likes
#4

assuming you have a table that stores all players’ data

local aBigDataTable = {}

when a player joins, you can get their data, decode it then put it into that table

local function loadPlayerData(player)
	local success, playerData = pcall(function()
		return yourdatastore:GetAsync("dataKey")
	end)
	if success then
		if playerData then
			playerData = httpService:JSONDecode(playerData)
		else
			playerData = {
				["XP"] = 0,
				["Health"] = 0,
				["HistoryOfSomething"] = {},
			}
		end
		aBigDataTable[player.Name] = playerData
	end
end

when you save you can encode the data, then save it

local function savePlayerData(playerName)
	local encodedData = httpService:JSONEncode(aBigDataTable[playerName])
	pcall(function()
		yourdatastore:SetAsync(encodedData)
	end)
end

depending on your use cases set async might not be the best, but I’d always recommend using pcalls for anything datastore related, you can also add retries if the request failed on both get and save

the data might look something like this when Player1 loads into the game

local aBigDataTable = {
	["Player1"] = {
		["XP"] = 0,
		["Health"] = 0,
		["HistoryOfSomething"] = {
			[1] = "added history maybe?",
			[2] = {
				["thisWorksToo"] = {}
			}
		}
	}
}
3 Likes
#6

How would I make it so it doesn’t reset the data for everyone? I use a different key for each data.

#7

you can use player’s ID as key

GetAsync(player.UserId)
#8

Alright, thank you! I will try it now.