Introducing MemoryStore SortedMap SortKey [Beta]

Updates Announcements
#8

This is so cool. Props to the engineers :smiley:

11 Likes
#9

thanks this greatly simplifies handling ELO. i spoke with a few developers that had not even realised memorystores can handle features like this because the key embedding was unintuitive and hacky.

11 Likes
#10

I definitely do hope that OrderedDataStores can catch up in the distant future. Data structures with sorted listing capabilities have always had very rigid requirements that required you to really butcher your schema in favour of sorting. Now for sorted maps you can just construct a key to be used internally for sorting without sacrificing the developer-facing query keys and writing weird code workarounds. Even better, it’s backwards compatible.

Unbelievably helpful.

12 Likes
#11

This makes storing data and sorting it much easier. Before, you would have two prefix the player’s score onto their UserId, or use the score as an index to the table to get the Name, UserId, etc… Now you can just put the key (UserId) in the key field and the score in the sort key field and away you go.

10 Likes
#12

This… is pretty nice!
Cant wait to use it.

7 Likes
#13

I’ve been saying we need some way to sort by value for years, including during the initial private memory stores beta, and that sorted maps have been extremely difficult to use. Introducing a SortKey fixes the biggest issue I’ve had with memory stores by far, and makes me want to try using them again.

Thank you for listening!!

8 Likes
#14

Wow, I think this is awesome. I had no idea you could accomplish such things with MemoryStore.
I might start using this for some stuff.

Also, I’ve seen someone use this to create a module, and I believe he will be pleased to learn about it.

5 Likes
#15

Where are the External Data Store storage stuff that you mentioned in a Dev Forum eariler this year about being able to access a players data from Google Excel sheet. I really want that,
being able to read and write to a Google Excel sheet sounds amazing.

2 Likes
#16

Hmm, a useful thing, but I don’t think many developers will use it

3 Likes
#17

Much needed update :+1:. Since you are advocating the use of leaderboards with MemoryStoreService, something that I may just not be understanding is if you had a Monthly leaderboard, you’d set the expiration to something like 2.628e+6, but perhaps your game has 1-5M MAU… We’d be cutting into the storage limits of the SortedMap, which means I’m assuming we’ll need to come up with some hacky way or cut off some of the deadweight after an update is finished (or at intervals)?

image
image743×162 15 KB

Also, I thought updating a key in a SortedMap also updates the expiration… In that case how would we be able to keep consistent Daily/Weekly/Monthly leaderboards? (Might be misunderstanding some documentation)

4 Likes
#18

Those are some good points :slight_smile:

For handling >1M players, we recommend sharding/partitioning your SortedMap. Due to the size limits on this data structure, it might be easier to use one SortedMap each for a daily/weekly leaderboard, and multiple SortedMaps for a monthly leaderboard.

For updating a key, you are right in that it updates the expiration with the supplied value. In case you want to track monthly leaderboards for the month of November for example, you would set expiration for an item being updated on November 3 to 30 - 3 = 27 days. You’d use a similar pattern for daily/weekly leaderboards :slight_smile:

9 Likes
#19

Will look into sharding/partitioning the monthly sorted map!

& Great solution! Should have thought of using a dynamic expiry haha. Great work and thanks!

2 Likes
#20

This is cool. I have use for it.

Some stuff you should add:

Multi-key sorting: Useful when sorting based on a single key isn’t sufficient.

Custom Sorting Callbacks: Ability for us (developers) to define a custom sorting method that could allow for more complex sorting logic outside of the realm of simple alphanumeric or numeric ordering.

Pagination support: Out-of-the-box pagination support that could help with large sets of data, especially if only a chunk / subset of data is in-use at any given time.

Range Queries: Retrieve a range of elements based on keys / sort keys. i.e fetching a range of scores from a leaderboard.

Bulk Operations: Often times, I will have a queue of things that I want to do apply to my dataset. I don’t want to apply them one by one. Especially for things that are basically the same. I would appreciate bulk-operation support features such as bulk-inserting, bulk-updating, bulk-deleting that would help me to interact with multiple elements in a single operation.

5 Likes
#21

please could someone explain this to me more simply - I do not entirely understand?
Is it like having set data in a key on the game which you can load onto the player?

1 Like
#22

Thanks for the feedback. We already support a few of them :slight_smile:

  • Multi-key sorting: The items in your Sorted Map will be first sorted by the sort key (if it exists) followed by the key. I’m assuming this is what you mean by multi-key sorting?

  • Range Queries: The GetRangeAsync endpoint supports exactly this behavior. You can specify a bound as a Lua table with one or more of sort key and key being provided.

3 Likes
#23

Hi,
I’ve done lots of testing and created a library (GitHub - arxkdev/Leaderboard: Leaderboard is an intuitive, open-source module designed to effortlessly establish and manage robust non-persistent & persistent leaderboards for your Roblox experiences.) meant for streamlining the creation of timed global leaderboards, but the limits for MemoryStoreService are just not feasible to be working with Global Leaderboards, most especially timed ones.

Sharding only works to a very limited extent where you wont have to worry about hitting the size limits, but you will have to worry about the request units you are using per minute across ALL the shards. Something I didn’t realize was that the request limit is subject to the whole experience, not individual maps. This makes working with sharding/partitioning near impossible with this type of use case. Let alone not having any way to request the budget so we cannot adjust our systems accordingly.

Request unit for all (experience) API calls: 1000 + 100 * [number of concurrent users]

Scenario:

Say you have 3 board types for a single stat: "Money"
Board 1: Daily - 1 Shard
Board 2: Weekly - 3 Shards
Board 3: Monthly - 5 Shards
For a total of 9 shards

1 CCU: this is 1,100 request units/minute for all API calls.
200 CCUs: this is 21,000 request units/minute for all API calls.

Average server size 8-20, say ours is 10.
20 Servers (200 CCUs / 10 server size), each server:

  • Runs every 90-120 seconds
    • UpdateFunction → Saves 10 UserIds (30 Request Units)
    • image
      image1046×131 9.47 KB
    • GetRangeAsync → Collects top 100 (900 Request Units)
    • We have 9 shards, that’s 900 request units for get and 30 request units for update/set.
  • 930 request units for that server
  • Out of all 20 servers, that’s 18,600 request units/minute

Keep in mind our threshold is 21,000 request units per minute.
You’re probably wondering, what If I wanted to add another leaderboard for say "Wins" or what if I want to add another board type, like Hourly?

Answer: You can’t. Do all those calculations and x2 it, 37,200 request units per minute for a threshold of 21,000 request units per minute. In fact, I would be doubtful you could create just ONE board without experiencing random internal errors and random miscalculated rate limiting (??)

This was mentioned in the documentation:

image
image624×637 70.5 KB

Why is it mentioned that Sharding helps with request limits? It does not. All it did was widespread the issue and solve usage limits.

Random errors I get 7/10 times using MSS (not sure if they’re related to the limits being hit or not):
image

TL;DR

  • MemoryStoreService with Global Leaderboards (or in some cases, as a whole) is extremely choppy
  • Why is Sharding/Partitioning recommended if all it does is solve map size limits and not also request limits for an experience?
  • A request budget feature similar to DatastoreService would be nice
  • MemoryStoreService API calls fail a lot, making it hard to stress-test, or test at all
  • MemoryStoreService API miscalculates the rate limits a lot. Again, a budget would help with debugging this
  • My library (above) is rendered useless if all it can support is 2 leaderboards with 2-3 board types in each without hitting limits
3 Likes
#24

Thank you for your work on this!

  • Sharding/Partitioning is recommended if you are hitting the per-data-structure request limits. If you run into universal/global limits, unfortunately sharding will not help. That limit is an absolute max for the Universe.
  • The failures for SortedMap.Update are being investigated. Could you share a bit more over DM about your use of UpdateAsync? Is all the code in the library that you linked?
1 Like
#25

:+1: Yeah, so perhaps the main takeaway here is using MemoryStoreService with Global Leaderboards (more than 1-2) isn’t really the best idea?

  • Correct me if I’m wrong but even with a single monolithic map being turned into shards would still have a better request limit than using individual shards. Because having say 10 shards would still have to make GetRangeAsync requests to all of it, and since it works in units, it would be a whopping 1,000 request units for all 10 shards, compared to one monolithic map being a single one time 100 request units but sacrificing that size limits issue.

  • Great to hear. Guilded profile or I can send you a message on the DevForum here, just let me know. All the code is in the library I linked. UpdateAsync is used with the UserId being the key, a transformer function which decompresses (logarithmic shortener) the previous value if there is one, returns nil if the old value is higher than the new value, then simply returns the compressed value and the new sort key. Do you think returning nil in the transformer could be causing internal parsing errors?

Some information about the Key modules inside of the library:

2 Likes
#26

Hello! I am a bit confused on this functionality. Maybe I am misunderstanding something, but for me I have found that the addition of a sortKey hinders the ability of some features in the GetRangeAsync.

Take this code for example:

local MemoryStoreService = game:GetService("MemoryStoreService")
local PlayersMap = MemoryStoreService:GetSortedMap("Players")

local USER_IDS = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
local COLORS = {"red", "green", "blue"}
local EXPIRATION = 10

for _, userId in USER_IDS do
	local randomColor = COLORS[math.random(1, #COLORS)]
	local key = randomColor.. "_".. userId
	local value = {} -- any data
	
	PlayersMap:SetAsync(key, value, EXPIRATION, userId) -- remove the `userId` parameter and the table will populate
end

----------

local color = "red"
local players = PlayersMap:GetRangeAsync(Enum.SortDirection.Ascending, 10, {key = color}, {key = color.. "__"})

print(`Players' data tagged with {color}:`, players)

On line 13, marked with a comment, the simple addition/removal of the optional sort key changes the results of the GetRangeAsync. When the sort key is present, the GetRangeAsync returns an empty table all the time, even though the lower and upper bounds of the function don’t mention the sort key at all. The docs of this function say that including the sort key in the bounds table is not necessary.

image
image915×372 48.7 KB

If I am misunderstanding this, would you mind explaining why the sort key is causing this issue?

Thank you!

1 Like
#27

Hello!

Your exclusiveLowerBound does not have a sortKey provided. When you provide a key as your exclusiveLowerBound, the GetRangeAsync API filters out all items with a sort key. This is because items with a sort key are sorted before items without a sort key.

Here’s an example

Key | Value  | SortKey
A      1        nil
B      2        a
C      3        1
D      4        2
E      5        nil

When sorted in ascending order, this is the order in which they are returned

C      3        1     -- items with numeric sort keys sort before items with string sort keys
D      4        2     -- 1 < 2, hence C sorts before D
B      2        a     -- items with string sort keys sort before items with no sort keys
A      1        nil   -- items with no sort keys are sorted last by key
E      5        nil   -- A < E, hence A sorts before E

When you make a request such as this
PlayersMap:GetRangeAsync(Enum.SortDirection.Ascending, 10, {key = A}, {key = Z.. "__"}), it will ignore the first 3 of the above 5 items, since they sort before item with key A.

In your case, it appears that your key is still what you want to sort on, since it is a composite key made up of the color and userId. You’d need to change your schema to use this new parameter.

One example would be if you change key to just userId, and sortKey to randomColor.. "_".. userId, then when you call GetRangeAsync, your exclusiveLowerBound can have nil key and color as the sort key.

Hope this helps!

1 Like