Will this code throttle datastore service?

Help and Feedback Code Review
, , ,
#1

I made this remote function where you can get the data of all factions as a client. I need this because I am making a faction system so the player can browse and join different factions. However, I am worried about the fact that I am making a datastore request and a groupservice request for each faction. What if there are 1,000 factions? I think that would overflow the datastore.

Here is my code:

FactionsRemotes:WaitForChild("GetAllFactionsData").OnServerInvoke = function(player)
	
	local returnTable = {}
	
	local regPages = FactionsDS:ListKeysAsync()
	
	local numFactions = 0
	
	while true do
		for _, datastoreKey in ipairs(regPages:GetCurrentPage()) do
-- GetFactionInfo makes 1 GetAsync request to a datastore.
			returnTable[datastoreKey.KeyName] = module.GetFactionInfo(tonumber(datastoreKey.KeyName)) 
			
			returnTable[datastoreKey.KeyName]["GroupInfo"] = GroupService:GetGroupInfoAsync(tonumber(datastoreKey.KeyName))
			
			numFactions += 1
		end
		

		if regPages.IsFinished  then -- or numFactions > 1000 here?
			break
		else
			regPages:AdvanceToNextPageAsync()
		end
	end
	
	return returnTable
end

My reference for this factions system is saber showdown. Is there a better way to get all factions without overflowing the datastore service? Any help would be appreciated.

1 Like
#2

It will throttle, and is also extremely vulnerable to any DOS attack. you should cache the results on the server and have the remote function just fetch those.

(an exploiter can spam fetch the remote function and throttle datastore requests, data for players won’t load or save)

1 Like
#3

So when the server starts, I call this function every 10 minutes or so? I think that’s a good idea for preventing a DOS attack, but I’m still concerned about the fact that it’s calling GetAsync on all keys at once. Is there a way to prevent that from throttling?

#4

an exponential backoff algorithm should be good enough, the loop should also wait an interval between every :AdvanceToNextPageAsync()

1 Like
#5

Sorry for my lack of knowledge, but how do I use the exponential backoff algorithim? I did some research and I’m assuming this is how I do it?

  1. Initial Attempt: Make a GetAsync request.
  2. First Failure: If it fails, wait for a fixed amount of time (let’s say x seconds).
  3. Retry 1: Make the request again.
  4. Second Failure: If it fails again, wait for double the previous waiting time (2x seconds).
  5. Retry 2: Make the request again.
  6. Third Failure: If it fails again, wait for double the previous waiting time (4x seconds).

If this is correct, what should x and the interval between each :AdvanceToNextPageAsync() be?

#6

heres a simple example for your use case:

function exponentialRetry(maxRetries: number, f: (...any) -> any, ...: any)
    local retryDelay = 1

    for i = 1, maxRetries do
        local success, result = pcall(f, ...)

        if (success) then return result end

        if i < maxRetries then
            task.wait(retryDelay)

            retryDelay *= 2
        end
    end
end

--in your case:
exponentialRetry(10, regPages.AdvanceToNextPageAsync, regPages)

--or if you're unfamiliar with oop implicit self:
exponentialRetry(10, function()
    regPages:AdvanceToNextPageAsync()
end)

note that this is just for retries, ontop of this you should still task.wait(0.1) between each AdvanceToNextPageAsync() to be safe

2 Likes
#7

I’m assuming I should also do exponential retry for module.GetFactionInfo() because it calls GetAsync?

#8

yeah you could, wouldn’t be a bad idea

1 Like
#9

I never thought I’d get so much help from devforum. Thank you so much!

1 Like