Introducing MemoryStore SortedMap SortKey [Beta]

Updates Announcements
#19

Will look into sharding/partitioning the monthly sorted map!

& Great solution! Should have thought of using a dynamic expiry haha. Great work and thanks!

2 Likes
#20

This is cool. I have use for it.

Some stuff you should add:

Multi-key sorting: Useful when sorting based on a single key isn’t sufficient.

Custom Sorting Callbacks: Ability for us (developers) to define a custom sorting method that could allow for more complex sorting logic outside of the realm of simple alphanumeric or numeric ordering.

Pagination support: Out-of-the-box pagination support that could help with large sets of data, especially if only a chunk / subset of data is in-use at any given time.

Range Queries: Retrieve a range of elements based on keys / sort keys. i.e fetching a range of scores from a leaderboard.

Bulk Operations: Often times, I will have a queue of things that I want to do apply to my dataset. I don’t want to apply them one by one. Especially for things that are basically the same. I would appreciate bulk-operation support features such as bulk-inserting, bulk-updating, bulk-deleting that would help me to interact with multiple elements in a single operation.

5 Likes
#21

please could someone explain this to me more simply - I do not entirely understand?
Is it like having set data in a key on the game which you can load onto the player?

1 Like
#22

Thanks for the feedback. We already support a few of them :slight_smile:

  • Multi-key sorting: The items in your Sorted Map will be first sorted by the sort key (if it exists) followed by the key. I’m assuming this is what you mean by multi-key sorting?

  • Range Queries: The GetRangeAsync endpoint supports exactly this behavior. You can specify a bound as a Lua table with one or more of sort key and key being provided.

3 Likes
#23

Hi,
I’ve done lots of testing and created a library (GitHub - arxkdev/Leaderboard: Leaderboard is an intuitive, open-source module designed to effortlessly establish and manage robust non-persistent & persistent leaderboards for your Roblox experiences.) meant for streamlining the creation of timed global leaderboards, but the limits for MemoryStoreService are just not feasible to be working with Global Leaderboards, most especially timed ones.

Sharding only works to a very limited extent where you wont have to worry about hitting the size limits, but you will have to worry about the request units you are using per minute across ALL the shards. Something I didn’t realize was that the request limit is subject to the whole experience, not individual maps. This makes working with sharding/partitioning near impossible with this type of use case. Let alone not having any way to request the budget so we cannot adjust our systems accordingly.

Request unit for all (experience) API calls: 1000 + 100 * [number of concurrent users]

Scenario:

Say you have 3 board types for a single stat: "Money"
Board 1: Daily - 1 Shard
Board 2: Weekly - 3 Shards
Board 3: Monthly - 5 Shards
For a total of 9 shards

1 CCU: this is 1,100 request units/minute for all API calls.
200 CCUs: this is 21,000 request units/minute for all API calls.

Average server size 8-20, say ours is 10.
20 Servers (200 CCUs / 10 server size), each server:

  • Runs every 90-120 seconds
    • UpdateFunction → Saves 10 UserIds (30 Request Units)
    • image
      image1046×131 9.47 KB
    • GetRangeAsync → Collects top 100 (900 Request Units)
    • We have 9 shards, that’s 900 request units for get and 30 request units for update/set.
  • 930 request units for that server
  • Out of all 20 servers, that’s 18,600 request units/minute

Keep in mind our threshold is 21,000 request units per minute.
You’re probably wondering, what If I wanted to add another leaderboard for say "Wins" or what if I want to add another board type, like Hourly?

Answer: You can’t. Do all those calculations and x2 it, 37,200 request units per minute for a threshold of 21,000 request units per minute. In fact, I would be doubtful you could create just ONE board without experiencing random internal errors and random miscalculated rate limiting (??)

This was mentioned in the documentation:

image
image624×637 70.5 KB

Why is it mentioned that Sharding helps with request limits? It does not. All it did was widespread the issue and solve usage limits.

Random errors I get 7/10 times using MSS (not sure if they’re related to the limits being hit or not):
image

TL;DR

  • MemoryStoreService with Global Leaderboards (or in some cases, as a whole) is extremely choppy
  • Why is Sharding/Partitioning recommended if all it does is solve map size limits and not also request limits for an experience?
  • A request budget feature similar to DatastoreService would be nice
  • MemoryStoreService API calls fail a lot, making it hard to stress-test, or test at all
  • MemoryStoreService API miscalculates the rate limits a lot. Again, a budget would help with debugging this
  • My library (above) is rendered useless if all it can support is 2 leaderboards with 2-3 board types in each without hitting limits
3 Likes
#24

Thank you for your work on this!

  • Sharding/Partitioning is recommended if you are hitting the per-data-structure request limits. If you run into universal/global limits, unfortunately sharding will not help. That limit is an absolute max for the Universe.
  • The failures for SortedMap.Update are being investigated. Could you share a bit more over DM about your use of UpdateAsync? Is all the code in the library that you linked?
1 Like
#25

:+1: Yeah, so perhaps the main takeaway here is using MemoryStoreService with Global Leaderboards (more than 1-2) isn’t really the best idea?

  • Correct me if I’m wrong but even with a single monolithic map being turned into shards would still have a better request limit than using individual shards. Because having say 10 shards would still have to make GetRangeAsync requests to all of it, and since it works in units, it would be a whopping 1,000 request units for all 10 shards, compared to one monolithic map being a single one time 100 request units but sacrificing that size limits issue.

  • Great to hear. Guilded profile or I can send you a message on the DevForum here, just let me know. All the code is in the library I linked. UpdateAsync is used with the UserId being the key, a transformer function which decompresses (logarithmic shortener) the previous value if there is one, returns nil if the old value is higher than the new value, then simply returns the compressed value and the new sort key. Do you think returning nil in the transformer could be causing internal parsing errors?

Some information about the Key modules inside of the library:

2 Likes
#26

Hello! I am a bit confused on this functionality. Maybe I am misunderstanding something, but for me I have found that the addition of a sortKey hinders the ability of some features in the GetRangeAsync.

Take this code for example:

local MemoryStoreService = game:GetService("MemoryStoreService")
local PlayersMap = MemoryStoreService:GetSortedMap("Players")

local USER_IDS = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
local COLORS = {"red", "green", "blue"}
local EXPIRATION = 10

for _, userId in USER_IDS do
	local randomColor = COLORS[math.random(1, #COLORS)]
	local key = randomColor.. "_".. userId
	local value = {} -- any data
	
	PlayersMap:SetAsync(key, value, EXPIRATION, userId) -- remove the `userId` parameter and the table will populate
end

----------

local color = "red"
local players = PlayersMap:GetRangeAsync(Enum.SortDirection.Ascending, 10, {key = color}, {key = color.. "__"})

print(`Players' data tagged with {color}:`, players)

On line 13, marked with a comment, the simple addition/removal of the optional sort key changes the results of the GetRangeAsync. When the sort key is present, the GetRangeAsync returns an empty table all the time, even though the lower and upper bounds of the function don’t mention the sort key at all. The docs of this function say that including the sort key in the bounds table is not necessary.

image
image915×372 48.7 KB

If I am misunderstanding this, would you mind explaining why the sort key is causing this issue?

Thank you!

1 Like
#27

Hello!

Your exclusiveLowerBound does not have a sortKey provided. When you provide a key as your exclusiveLowerBound, the GetRangeAsync API filters out all items with a sort key. This is because items with a sort key are sorted before items without a sort key.

Here’s an example

Key | Value  | SortKey
A      1        nil
B      2        a
C      3        1
D      4        2
E      5        nil

When sorted in ascending order, this is the order in which they are returned

C      3        1     -- items with numeric sort keys sort before items with string sort keys
D      4        2     -- 1 < 2, hence C sorts before D
B      2        a     -- items with string sort keys sort before items with no sort keys
A      1        nil   -- items with no sort keys are sorted last by key
E      5        nil   -- A < E, hence A sorts before E

When you make a request such as this
PlayersMap:GetRangeAsync(Enum.SortDirection.Ascending, 10, {key = A}, {key = Z.. "__"}), it will ignore the first 3 of the above 5 items, since they sort before item with key A.

In your case, it appears that your key is still what you want to sort on, since it is a composite key made up of the color and userId. You’d need to change your schema to use this new parameter.

One example would be if you change key to just userId, and sortKey to randomColor.. "_".. userId, then when you call GetRangeAsync, your exclusiveLowerBound can have nil key and color as the sort key.

Hope this helps!

1 Like
#28

Thank you for the response! I updated my test code with your example (reproduced below) but still found that the table wouldn’t populate.

local MemoryStoreService = game:GetService("MemoryStoreService")
local PlayersMap = MemoryStoreService:GetSortedMap("Players")

local USER_IDS = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
local COLORS = {"red", "green", "blue"}
local EXPIRATION = 10

for _, userId in USER_IDS do
	local randomColor = COLORS[math.random(1, #COLORS)]
	local sortKey = randomColor.. "_".. userId
	local value = {} -- any data
	
	print(userId, sortKey)
	PlayersMap:SetAsync(userId, value, EXPIRATION, sortKey)
end

----------

local color = "red"
local players = PlayersMap:GetRangeAsync(Enum.SortDirection.Ascending, 10, {sortKey = color}, {sortKey = color.. "_~"}) -- tilde has a high utf8 decimal

print(`Players' data tagged with {color}:`, players)

Based on your order example, I believe this is what should be happening in the above code:

Key | Value  | SortKey
1      {}       red_1
2      {}       blue_2
3      {}       red_3
4      {}       green_4
5      {}       red_5

And thus should sort like this when exclusiveLowerBound = {sortKey = "red"} and exclusiveUpperBound = {sortKey = "red_~"}:

Key | Value  | SortKey
1      {}       red_1
3      {}       red_3
5      {}       red_5
-- these shouldn't be returned since its exclusive🔽
2      {}       blue_2
4      {}       green_4
1 Like
#29

Awesome update, I will definitely use it to optimize my matchmaking even more.

#30

So when the sort key is present in the data, do the exclusiveLowerBound and exclusiveUpperBound even take into account the key?

Take this sample for example:

local MemoryStoreService = game:GetService("MemoryStoreService")
local PlayersMap = MemoryStoreService:GetSortedMap("Players")

local USER_IDS = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
local EXPIRATION = 10

local function padNumber(n: number, c: number, decimal: boolean)
	return string.format(`%0.{c}{if decimal then "f" else "i"}`, n)
end

for i, userId in USER_IDS do
	local sortKey = padNumber(math.random(), 3, true)
	local value = {} -- any data

	PlayersMap:SetAsync(padNumber(userId, 2), value, EXPIRATION, sortKey)
end

-- Get all user IDs between 3 and 7 (exclusive)
local lowerBound = {key = "03", sortKey = padNumber(0, 3, true)}
local upperBound = {key = "07", sortKey = padNumber(1, 3, true)}
local result = PlayersMap:GetRangeAsync(Enum.SortDirection.Ascending, 10, lowerBound, upperBound)

print(result) -- prints everything, ignores key entirely?

-- Get all percentages between 0.5 and 1
local lowerBound2 = {key = "00", sortKey = padNumber(0.5, 3, true)}
local upperBound2 = {key = "11", sortKey = padNumber(1, 3, true)}
local result2 = PlayersMap:GetRangeAsync(Enum.SortDirection.Ascending, 10, lowerBound2, upperBound2)

print(result2) -- works as intended, but try setting the key to "11" and it doesn't work as expected

I would expect the data to be returned be 04, 05, 06 user IDs, but instead it returns every item, as if the key is useless in the search and it only takes into account the sortKey. I understand I could probably make this example work by swapping the key with the sort key, but wouldn’t the expected behavior be for it to work both ways? What if I wanted to search range by key and sort key separately, like in the example above where I want to get a range of user IDs and a range of percentages? In both cases, presumably because I have a sort key set, the key does nothing in the GetRangeAsync() function.

#31

Both of your code snippets are working as designed :slight_smile:
To clarify, the bounds are exclusive. In your first snippet (the one with the colors), when your lower bound has a sortKey = “red”, it will ignore all items with red as the sortKey.
The 5 items will sort like so

Key   |   SortKey
2         blue_2
4         green_4
1         red_1
3         red_3
5         red_5

Entries are sorted first by sortKey (alphabetically, since these are string sort keys) and then by key.

Now say you care about getting all entries with sortKey “red”, you can get them via a call like this

local players = PlayersMap:GetRangeAsync(Enum.SortDirection.Ascending, 10, { sortKey = 'green' }, { sortKey = 'red' .. '_~' }) -- tilde has a high utf8 decimal

If the lower bound is set to red, all entries with red as the sort key will be excluded from the results since sort key is exclusive.

In your second example, it appears as if the keys are being ignored, since the sortKey bounds encompass all items. The way the GetRangeAsync call works is -

  • Check if the sort Key for the entry is > the lower bound sort key. If it is, it will be included in the results regardless of the key. Same for the upper bound
  • If the sort key for the entry is equal to the sort key of the lower/upper bound, it will then compare the key with the key provided as part of the bound
2 Likes
#32

Would there be any latency or drawbacks for repeatedly calling GetSortedMap instead of storing a fetched map in a table?

#33

No there won’t be any drawbacks of getting a sorted map in your script. Could you elaborate what you mean by repeatedly calling this? In most scenarios, you’d need to call GetSortedMap to get the sorted map by name, and then use the API methods you require for your use case in your script(s).

#34

Apologies, I thought that fetching MemoryStores was rate-limited and thus had to be cached with a single call instance as such;

local map = game:GetService("MemoryStoreService");
local cache = {}

local function loadMap(name: string)
	local cached = cache[name]
	if not cached then
		cache[name] = map:GetSortedMap(name)
	end
	return cache[name]
end
#35

Is there an ETA on when MemoryStoreHashMap will be released?

1 Like
#36

Hello! Yes the program is in private beta right now, with plans of releasing it this quarter. We plan to open the beta to the public very shortly. Stay tuned!

2 Likes
#37

Beta is now live! MemoryStore HashMap [Beta]

1 Like
#38

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.