Is it safe to iterate through a DataStore on a large scale like this?

Help and Feedback Scripting Support
,
#1

Creating a guild/clan system and I was wondering if it would be safe to iterate through a datastore using :GetAsync() like this (code works, was just wondering about the practicality):

local pages = DataStore:ListKeysAsync()

while true do
    local items = pages:GetCurrentPage()
    for _, v in ipairs(items) do
        local value = DataStore:GetAsync(v.KeyName)
        self.CachedGuilds[v.KeyName] = value
    end
    if pages.IsFinished then
        break
    end
    pages:AdvanceToNextPageAsync()
end

The purpose of this is to create a cache on the server containing a list of all globally existing guilds, which could then be further used for things like guild leaderboard rankings, or guild join list. Guild data is saved in it’s own DataStore, separate from player data, but I’m still worried about how safe this would be if iterating through hundreds of guilds.

#2

In my opinion, probably not very safe. You only get 60 + numPlayers × 10 GetAsync calls to all data stores per minute (source) so you’d hit your limit very quickly; after only about 70 guilds on server startup assuming there’s 1 person in a server. Doing it this way also takes away from how many players’ data you can load within one minute, even if guild data and player data are separate so you will almost definitely run into players complaining that their data taking a long time to load, or just not loading at all.

It’s ultimately up to you but those are my two cents.

1 Like
#3

Thanks, that formula is nice to know. I’m assuming thats applied per server and not globally right?

I tried running a stress test last night and noticed it was :ListKeysAsync() being problematic, not :GetAsync(). I guess the safest thing to do would be to not implement a guild leaderboard or guild join list but rather provide a direct guild search feature to access a single guild value at a time.

#4

Yes, per server.

If :ListKeysAsync is being slow, :GetAsync will be slow as well since you are relying on :ListKeysAsync to call :GetAsync.

And yes, the best practice would likely be to have a search feature, or use an external dependency that allows for leaderboard listings for guilds. I don’t have any on hand so can’t make any recommendations unfortunately.

1 Like
#5

The pages item only consists of 1 page everytime, you should probably use the cursor argument of ListKeysAsync(), but I am not sure how that works myself yet.