DataStore requests are being incorrectly queued

While shutting down servers in my game, I ran into this issue:

DataStore save requests were being added to a queue, and eventually, they timed out and the server shut down before my data could even be saved!

Why are DataStore requests being queued when there is plenty of budget for them?

This is because you trying to update the same key very quickly. You are only able to update the same key every 6 seconds.

See the datastore limits here:
https://developer.roblox.com/articles/Datastore-Errors

1 Like

For this and other details, also refer to the budgeting/limits sections in:

The simplest way to prevent this issue from occurring is to have your data handler track so-called dirty keys: keys that have data that hasn’t been saved yet. Only save dirty keys on server shutdown, and you don’t have to save non-dirty keys in your autosaving loop.

That’s the thing though, I’m saving each player’s save file once. I’m not trying to update the same key more than once, much less 10+ times.

Save player data:

local function __intUpdatePlayerSaveFileData(player, saveFileNumber, playerSaveFileData)
	local PlayerId 			= player.userId
	local Slot 				= saveFileNumber
	local TimeStamp 		= os.time()

		
	local Success, Error = pcall(function()
		
		local Suffix 		= "-slot"..tostring(Slot)
		local OrderedStore 	= game:GetService("DataStoreService"):GetOrderedDataStore(tostring(PlayerId), "PlayerSaveTimes" .. datastoreVersion .. Suffix)
		local DataStore 	= game:GetService("DataStoreService"):GetDataStore(tostring(PlayerId), "PlayerData" .. datastoreVersion .. Suffix)
		
		DataStore:SetAsync(tostring(TimeStamp), playerSaveFileData)
		OrderedStore:SetAsync("s" .. tostring(TimeStamp), TimeStamp)
	end)
		
	return Success, Error, TimeStamp
end

On player removing:

local function onPlayerRemoving(player)
	if playerDataContainer[player] then
		-- attempt to retry up to 3 times on failure
		for i = 1, 3 do
			local Success, Error, TimeStamp = datastoreInterface:updatePlayerSaveFileData(player, playerDataContainer[player])
			if not Success then
				warn(player.Name,"'s data failed to save.",Error)
				network:invoke("reportError", player, "error", "Failed to save player data: "..Error)
				network:invoke("reportAnalyticsEvent",player,"data:fail:save")				
			else
				playerDataContainer[player] = nil
				return TimeStamp
			end
		end
	end
end

BindToClose:

game:BindToClose(function()
	
	shuttingDown = true

	local msg = Instance.new("Message")
	msg.Text = "Servers are shutting down for a Vesteria or Roblox client update. Your data is now saving..."
	msg.Parent = workspace
	
	if game:GetService("RunService"):IsStudio() then return end
	local playersToSave = {}
	local playerCount = 0
	for i,player in pairs(game.Players:GetPlayers()) do
		local playerName = player.Name
		playersToSave[playerName] = player
		playerCount = playerCount + 1
		spawn(function()
			local success, err = pcall(onPlayerRemoving,player)
			if success then
				playersToSave[playerName] = nil
				playerCount = playerCount - 1
			else
				warn("Failed to save",playerName,"data")
				warn(err)
			end
		end)
	end
	repeat wait(0.1) until playerCount <= 0
end)

How does this code save the same key 10 times?

DataStores aren’t accessed anywhere else in my game.

2 Likes

Every player has the same key (the timestamp) being saved. However, it is being saved to a different DataStore for every player. Perhaps these requests are being incorrectly grouped?

Is it possible that onPlayerRemoving is called twice for each player in quick succession because they are both leaving the game and the OnClose call is happening? That could explain why you would get the warnings, i.e. one for each player since two calls are happening

2 Likes

I know it’s hacky but try spacing your datastore requests with yields. For example,

DataStore:SetAsync(key, value)
wait()
DataStore:SetAsync(key, value)
1 Like

OnPlayerRemoving nukes all references to the player’s data after it is successful, so it can’t be called again after a successful save.

There could be an issue if someone left the moment BindToClose got fired, but it was a small server and no one left the server when this occurred, since they were waiting for the reconnect prompt. All players are kicked at the same time, once everyone’s data has successfully saved or the BindToClose call times out.

1 Like

To ensure that OnPlayerRemoving is not called twice you could set playerDataContainer[player] to nil when it is first called and just use a local variable to keep the players data after that point. I don’t think there is any way that the key throttling could be accidentally happening for keys named the same in different datastores. I think we should differentiate these error messages to make situations like this easier to figure out.

To give you a feature request to work with:
https://devforum.roblox.com/t/datastores-improve-clarity-of-throttling-warnings/175852

(Note that throttle warning was updated to what is in OP here since I filed that thread)

1 Like

You should not be doing two data store cals within the same pcall block. If the second fails you are duplicating the first one and I have no idea what effect this would cause in your save system.

I would split these up and only process the ones that fail. You could even create a filed data store call list which can be processed after your main save loop?

Edit
Which one of your data store calls fails?

There would be no effect. The second call is esentially a reference (timestamp) to the first call. If the second one fails, the first will never be accessed.

	DataStore:SetAsync(tostring(TimeStamp), playerSaveFileData)
	OrderedStore:SetAsync("s" .. tostring(TimeStamp), TimeStamp)
1 Like

if the second fails you have no delay. The next request will fail in all cases as you are trying to set the same key within 6 seconds which would explain your warning?

If a request fails, it prints to output. None of the requests failed in the original log.

1 Like

I see what you mean. Is this something you can replicate?

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.