Architecture for creating a Daily Leaderboard with Datastores or Memory Storage

Help and Feedback Scripting Support
,
1

Hi all,

I’m trying to make a daily leaderboard, but I’m stuck on the correct architecture and hoping to get some feedback from more experienced devs. It’s for a parkour-type game for which I want to make a Top 5 of the fastest times of the day.

Attempt 1 - Ordered DataStore
At first, I tried using an OrderedDataStore. Which made sense, I put the userID as a Key and the Time as a Value. The OrderedDataStore then nicely ordered the times from fastest to slowest, so I just need to get the top 5, and I’m done.
BUT, I wanted only times for today. So in my first version, I just put the date in the name of the OrderedDataStore like such:

local DataStoreService = game:GetService("DataStoreService")
local LeaderboardStore = DataStoreService:GetOrderedDataStore("DailyLeaderboard" .. DATE)

But this would create a backlog of OrderedDatastores from previous days, and there is no API/Function to delete an old DataStore. (As far as I found in the documentation, at least). So I abandoned this as it didn’t feel like a clean architecture to pile up unused DataStores. It’s also explicitly mentioned in the documentation that you should limit the number of tables.

Attempt 2 - SortedMap in MemoryStore
So then I learned about the MemoryStore, it had an expiration parameter, so I thought, “perfect”! I just create a function that calculates the seconds left of today, put it as an expiration period, and all the daily entries expire at midnight, which would give me a clean Leaderboard every time at midnight.
But, the sorted map of a Memorystore sorts on the key instead of the value. Although the documentation says the key needs to be a string, I managed to store the number of seconds as an integer, and it works and sorts fine. I just put the userID as a value. And again, I have a sorted top 5 of the fastest times.
But it created two new issues.

  1. Because the UserId is now the value, it would allow players to get multiple entries in the daily leaderboard. I fixed this with a secondary DataStore that stores the last fastest time of a given userID. So that way, I can check, was the player quicker than last his/her last attempt? If so, add a new entry and delete the old one. So that issue I managed to fix. (Although now that I’m writing it, I would still need to reset the secondary datastore at midnight, but I think I could fix that somehow)
  2. If people finish on the same time mark in seconds. For example, if two players finished in 8 seconds, the second player wouldn’t be able to store his time as the unique key of 8 seconds already exists. And I didn’t figure out how to resolve it. You could add the user_id in the key, but sorting numbers alphabetically doesn’t work with numbers like 1, 11, 2, 3

Attempt 3 - Prefixes
Then I read in the documentation of DataStores again that you can add prefixes. So I thought ok, let’s add a prefix of the date to the user ID key. So it would be, for example, 26072023/userid1 as a key. When I want to display the top five, I filter on the prefix of today, and I have a top 5 again. I would then need to create a throttled separate thread to delete the old data record by record with a 0.10 wait time in between. Not super great, but I would have a DataStore and no old irrelevant data.
BUT in the documentation of the OrderedDataStore there are no mentions of prefixes, and the function “ListKeysAsync” to get a list of keys with a prefix doesn’t exist for OrderedDataStores. I wrongly assumed it would be inherited from GlobalDataStore. But it’s only part of the regular DataStore.

Attempt 4 - Full manual approach
Another way I thought as an option, was a full manual approach. Didn’t implement it yet. But the concept was when a player finishes the parkour, it would fetch a latest top 5, manually add the new entry if it was faster, do a sort and store the end result again, for example as a JSON table or something.
But when I gave it more thought it could create conditions. For example, when two players finish at the same time. They both get the latest top 5, do their adding and sorting and depending how all those async processes go, the second player could overwrite the data of the first.
(Although less likely than duplicate keys in the memory store, it doesn’t feel right, and there should be a better/simpler method.)

And now I’m out of ideas for how I should approach this. :frowning: #PleaseHelp :pray:

2

For future readers, I ended up with the following architecture.

  • When a player finishes a parkour, I put the results in a MemoryStoreQueue (MemoryStoreQueue | Documentation - Roblox Creator Hub)
    => This resolves the race condition if two players finish simultaneously.

  • Then I have a MemoryStore in which I store a Table of the top 5 with expiration at midnight by calculating the number of seconds left for today. This way, the leaderboard is refreshed daily.

  • I then have a coroutine that reads from the MemoryStoreQueue, and if a new entry was placed in the queue, fetches the last top 5 and updates it if needed.

There could still be an issue if two servers pick up the same or different records simultaneously from the queue. So it would be even more solid if only one server could read from the Queue and process it. But I haven’t seen a way to limit a coroutine to a single server. So would still appreciate help/feedback on that part :slight_smile:

3

Typically you would have a “leader” server doing the calculations. You can store the leader in a different memory store. If there is no leader then the current server becomes a leader. You can refresh the leader every 1,5,60 minutes.

Now, for you race condition and memory queue, I think that’s unnecessarily complicated. If you just used UpdateAsync on the memory store, there cannot be race conditions. On UpdateAsync only one server performs the operation on the memory store. If two servers were to call it at once, then one server would perform the update first; the other server would block until the first one finishes and then it would get an updated version on which to perform its calculation.

Overall, I found memory store queues useless in most scenarios.

PS: Your first approach with multiple datastores per day is completely fine. There is absolutely no issue in creating multiple tables. But memory stores are just faster, so it’s better to do that for short lived data.