Introducing MemoryStore SortedMap SortKey [Beta]

Hi Developers,

We are excited to announce the Sorted Map Data Structure in the MemoryStore Service now supports sorting the items with a special field, in addition to the key! This feature is fully backwards compatible and will not disrupt your existing usage of MemoryStore Sorted Maps.

We understand that for certain use cases, such as leaderboards, you would like to have the ability to sort items in your Sorted Map by something other than the key. While this can be done by prefixing the key with a value, you still need to think about cases such as padding a number with leading 0s.

With this new sort key field, the data in your Sorted Map will be sorted using this optionally supplied sort key. Some details on this parameter:

  • The sort key can be up to 128 bytes in size
  • It can be a string or a number (integer or floating point)

Items with a sort key will sort before items without a sort key. Numeric sort keys will take precedence over items with string sort keys.

The existing APIs supported for the Sorted Map data structure have been updated to support this new field.

  • GetAsync: Returns sort key (if present) along with value
  • SetAsync: Accepts an optional sort key along with key, value and expiration
  • UpdateAsync: In addition to the value, provides the sort key to the transform callback and accepts the sort key from the transform callback
  • GetRangeAsync: Accepts a Lua table as a boundary instead of a string key, where one or both of key or sort key are provided in the following format
{
  sortKey: Variant,
  key: string,
}

Each returned item will have a sort key (if present) along with the value.

For example, here are some items in a sorted map in ascending order

{Key: "player1", Value: v, SortKey: -1}                 -- negative sort key
{Key: "player2", Value: v, SortKey: 0}                  -- 0 sort key
{Key: "player4", Value: v, SortKey: 1}                  -- 1 sort key, key "player4"
{Key: "player5", Value: v, SortKey: 1}                  -- 1 sort key, key "player5"
{Key: "player3", Value: v, SortKey: 3.14}               -- 3.14 sort key
{Key: "player6", Value: v, SortKey: "someString"}       -- "someString" sort key
{Key: "player0", Value: v}                              -- no sort key, key "player0"
{Key: "player7", Value: v}                              -- no sort key, key "player7"

Here is an example of how this feature can be used to build a leaderboard:

-- Updating the leaderboard score for a player in a Sorted Map

local MemoryStoreService = game:GetService("MemoryStoreService")
local sortedMap = MemoryStoreService:GetSortedMap("Leaderboard")

local function updateLeaderboard(itemKey, killsToAdd, deathsToAdd)
  local score = killsToAdd / math.max(deathsToAdd, 1)
  local success, newStats, newScore = pcall(function()
    return sortedMap:UpdateAsync(itemKey, function(playerStats, playerScore)
      playerStats = playerStats or { kills = 0, deaths = 0 }
      playerStats.kills += killsToAdd
      playerStats.deaths += deathsToAdd
      if playerStats then
        -- `playerScore` is the sortKey being used to sort items in the map
        playerScore = playerStats.kills / math.max(playerStats.deaths, 1)
        return playerStats, playerScore
      end
      return nil
    end, 30)
  end)
  if success then
    print(newStats)
    print(newScore)
  end
end

To learn more about this new field, check out the new API documentation and updated API guide available now.

As always, your feedback is invaluable for us to improve our services and fit your needs. Feel free to share any comments or questions below, and we’ll do our best to answer. We can’t wait to see what you build with this new feature!

Happy building!

The Roblox Creator Services Team

186 Likes

This topic was automatically opened after 10 minutes.

I love this new update. This will really help me.

17 Likes

seems interesting gonna try it eventually

16 Likes

Great to see improvements to MemoryStores! Will see if I can make this a viable replacement for OrderedDatastores which aren’t super great for frequently updating data.

15 Likes

Updates like this, where a bunch of ways to use it immediately pop into my head, are the best type of updates. :slight_smile:

13 Likes

One feature request I have (I’ve wanted this for OrderedDatastores but it has never been implemented) is the ability to provide a key and have the API return that key’s position.

If this is something that is technically possible, it would be super helpful for some of the projects I am working on.

19 Likes

This is so cool. Props to the engineers :smiley:

10 Likes

thanks this greatly simplifies handling ELO. i spoke with a few developers that had not even realised memorystores can handle features like this because the key embedding was unintuitive and hacky.

10 Likes

I definitely do hope that OrderedDataStores can catch up in the distant future. Data structures with sorted listing capabilities have always had very rigid requirements that required you to really butcher your schema in favour of sorting. Now for sorted maps you can just construct a key to be used internally for sorting without sacrificing the developer-facing query keys and writing weird code workarounds. Even better, it’s backwards compatible.

Unbelievably helpful.

11 Likes

This makes storing data and sorting it much easier. Before, you would have two prefix the player’s score onto their UserId, or use the score as an index to the table to get the Name, UserId, etc… Now you can just put the key (UserId) in the key field and the score in the sort key field and away you go.

9 Likes

This… is pretty nice!
Cant wait to use it.

7 Likes

I’ve been saying we need some way to sort by value for years, including during the initial private memory stores beta, and that sorted maps have been extremely difficult to use. Introducing a SortKey fixes the biggest issue I’ve had with memory stores by far, and makes me want to try using them again.

Thank you for listening!!

8 Likes

Wow, I think this is awesome. I had no idea you could accomplish such things with MemoryStore.
I might start using this for some stuff.

Also, I’ve seen someone use this to create a module, and I believe he will be pleased to learn about it.

5 Likes

Where are the External Data Store storage stuff that you mentioned in a Dev Forum eariler this year about being able to access a players data from Google Excel sheet. I really want that,
being able to read and write to a Google Excel sheet sounds amazing.

2 Likes

Hmm, a useful thing, but I don’t think many developers will use it

3 Likes

Much needed update :+1:. Since you are advocating the use of leaderboards with MemoryStoreService, something that I may just not be understanding is if you had a Monthly leaderboard, you’d set the expiration to something like 2.628e+6, but perhaps your game has 1-5M MAU… We’d be cutting into the storage limits of the SortedMap, which means I’m assuming we’ll need to come up with some hacky way or cut off some of the deadweight after an update is finished (or at intervals)?

Also, I thought updating a key in a SortedMap also updates the expiration… In that case how would we be able to keep consistent Daily/Weekly/Monthly leaderboards? (Might be misunderstanding some documentation)

4 Likes

Those are some good points :slight_smile:

For handling >1M players, we recommend sharding/partitioning your SortedMap. Due to the size limits on this data structure, it might be easier to use one SortedMap each for a daily/weekly leaderboard, and multiple SortedMaps for a monthly leaderboard.

For updating a key, you are right in that it updates the expiration with the supplied value. In case you want to track monthly leaderboards for the month of November for example, you would set expiration for an item being updated on November 3 to 30 - 3 = 27 days. You’d use a similar pattern for daily/weekly leaderboards :slight_smile:

9 Likes

Will look into sharding/partitioning the monthly sorted map!

& Great solution! Should have thought of using a dynamic expiry haha. Great work and thanks!

2 Likes

This is cool. I have use for it.

Some stuff you should add:

Multi-key sorting: Useful when sorting based on a single key isn’t sufficient.

Custom Sorting Callbacks: Ability for us (developers) to define a custom sorting method that could allow for more complex sorting logic outside of the realm of simple alphanumeric or numeric ordering.

Pagination support: Out-of-the-box pagination support that could help with large sets of data, especially if only a chunk / subset of data is in-use at any given time.

Range Queries: Retrieve a range of elements based on keys / sort keys. i.e fetching a range of scores from a leaderboard.

Bulk Operations: Often times, I will have a queue of things that I want to do apply to my dataset. I don’t want to apply them one by one. Especially for things that are basically the same. I would appreciate bulk-operation support features such as bulk-inserting, bulk-updating, bulk-deleting that would help me to interact with multiple elements in a single operation.

5 Likes