New Data Store GetKeysAsync()

[M0RGOTH (LordEarendil)]

I was trying to find similar requests/questions but all I could find were these:

http://devforum.roblox.com/t/data-store-transactions/22177

DataStoreService:GetDataStores() and DataStore:GetKeys()

We pretty much already have this, but in a slightly less obvious format. Whenever you create a datastore add a value to an ordered data store named “Datastores”. The key would be your datastore’s name and the value would be whatever. Whenever you create a key in your datastore add a key in an ordered data store named datastorename…“Keys”. The key would be the key and the value would be whatever. Now you can use GetSortedAsync in your ordered data store to retrieve the list in a clean and sc…

Neither of these actually addressed the issue I have.

My game involves multiple levels, so each player’s data is stored as a table which contains an array of all of the data that a player has for every level. This means I can load the data for every level for a player with only one GetAsync. I also keep an ODS for each level containing a single value which denotes a high score. This value is replicated within each player’s table (reason for this described below).

I am building a per-level leaderboard system which gives the user the option to view an all-time leaderboard, an in-server leaderboard, and a friends leaderboard. The all-time is quite easy, I pull from the ODS for each level. The in-server is even easier, I have all the player data for the players currently in the server pre-loaded. The friends leaderboard still isn’t too bad. I use Players.GetFriendsAsync to iterate through all of the friends of a player. Then I do a single GetAsync for each friend to load their data (unless they are already in the server). And now I can display it on the client.

So what’s the problem? Well, a player may have up to 200 friends and it is possible none of them are in their server. If this is the case, I will need to do 200 GetAsyncs at once. This is on top of any loading of player data whenever a player joins the server. Now, I currently cache each leaderboard for a set period of time. The all-time leaderboard is currently cached for a minute. The in-server leaderboard isn’t cached at all since the data is loaded and up to date at all times. Each player’s friends leaderboard is cached for 2 minutes. Each of these leaderboards is not updated unless their data has expired and a player actually requests it (opens the leaderboard menu in the lobby).

Since I have a limit of 60 + 10*numPlayers of GetAsync per minute, this may be just feasible if there are more than 4 players in the server at a time. And even then, it’s a stretch. I don’t want to risk a ton of (more important, like loading player data) data store failure because of my leaderboard, however this leaderboard is an important part of my game.

If there is a better way I could go about this, please let me know. I am open to suggestions. However, I think this is a good place to think about a new API:

<array> values GlobalDataStore.GetKeysAsync(<array> keys)

Now, I don’t know anything about the actual ROBLOX-side implementation of data stores, but I presume this would actually be a lot cheaper for ROBLOX than a series of individual GetAsyncs. As well, it is much more reliable and easy to handle errors for the developer, since there would be a single point of failure rather than several. Lastly, it would take less time, since there would only be one network request, rather than several. Of course, the max length of the keys array would be determined by ROBLOX. I would be happy if it was even as low as 20, but I think it would be fantastic if it was as large as 200. At any rate, I can see how this would be very useful for many developers and I think it is a necessary improvement to the overall system.

[Polymorphic]

Support, though I imagine ROBLOX will want to do this with their pages API.

I would presume its rather expensive to transmit so much information at a time and it wouldn’t be scalable to submit upwards to millions of keys to a game (let alone allocate the memory for it when we’re probably only interested in a few).

So, rather than returning an array I suggest it would return a modified page object.

I would also appreciate a way to fetch keys (completely different feature request) but would be implemented in a similar way.

[Baumz]

Instead of pulling the score from the Friends data, why not pull it from the ODS? Just match the Friends’ Id with the Id in the ODS to get their score. Or does the ODS only record the top X amount, so their friends might not be in it?

[M0RGOTH (LordEarendil)]

Well, there is an ODS for each level. So if I did that, I would have to do levelCount*friendCount GetAsyncs instead of just friendCount GetAsyncs

[Baumz]

Don’t you pull data from ODS anyways?

1. You call GetAsync on the ODS once for each Level and store/cache it.
2. You match the Player’s Id in each ODS to get their score for each level.
3. Then loop through the Player’s friends matching each one to each ODS to get their score.

In the end you would only have to make one datastore call for each level.

[M0RGOTH (LordEarendil)]

I only pull the top 100 from ODS. It could theoretically have millions of entries, could it not? I don’t think it makes sense to load everything from an ODS that could get that big.

[M0RGOTH (LordEarendil)]

Plus, there is an ODS for each level. So that becomes every crazier. That would be a huge memory hog if nothing else.