Analytics displays the game as being orders of magnitude below request unit quota, also far below the limits for an individual key, and yet the game is regularly getting throttled during player spikes preventing matchmaking from functioning as expected.
Either analytics are showing the data or the quota incorrectly, there is a bug with the throttling, or there is some undocumented limit to MemoryStore requests I am unaware of.
Attached are graphs showing the increase in MemoryStore errors during player spikes, despite not being close to quota according to analytics.
The ID of the specific experience this is occurring for (though potentially others also) is 3929033413.
The only relevant documented MemoryStore limits are:
100,000 requests per minute universe wide for a single key, which I can’t be hitting as my total requests per minute rarely exceeds 25,000.
1000 + 100 * [number of concurrent users] request units per minute, which per analytics I am orders of magnitude below.
Is there any updates? I am using memory store for session lock in my game, and some of players are getting stuck in the loading screen due to the last server throttled the memory store requests
@ivorycastle Since you responded here and seem to work in the Memory Stores area, could we get an update on this? I’m really unsure why this is happening. Is there a per-server throttle for MemoryStoreService SortedMaps that isn’t documented?
@Burgundy2014 It’s been nearly 3 months since you said you were going to look into this. Can we get an update or will we continue to be ignored?
Sorry about the delay. We have in fact been working on this. There are have been a few Memory Store quota limits tickets and issue, so we have been trying to disambiguate the problems from each other.
I want to clarify the problem that this post is for.
My understanding is that it is the experience-wide API request limit that is being hit:
1000 + 100 * [number of concurrent users] request units per minute
Is this only happening for small games with few concurrent users? Can you let me know your experiences user ID and average concurrent users?
If the above is the issue, we believe we understand the cause and are exploring options to fix it. The issue is finding the most accurate way to share a global quota in a distributed system. We though our original system was effective and have already tweaked it to solve an earlier issue, but we may need a larger scale redesign.
Thank you for your response—I appreciate your efforts. My experience ID is 6321891709.
Reflecting on this, it seems possible. In my scenario, I was invoking :GetRangeAsync on 9 different SortedMaps in each server every 60 seconds, primarily to refresh leaderboards every minute. Initially when the problem begun I changed the refresh time to 2 minutes, then 3, then 4 and 5, and this still did not resolve the issue. As problems persisted despite my efforts to resolve them, I shifted to using my own external API. The API retrieves SortedMap data from https://apis.roblox.com/cloud/v2/universes/UNIVERSEID/memory-store/sorted-maps/leaderboard/items every 60 seconds, caches it, and then my game servers request it via HttpService.
However, this raises questions about the accuracy of the Memory Store observability dashboard. The issue is either:
I am being rate-limited too frequently than the documented rate limits.
OR
The rate limiting is accurate, but the MemoryStore observability dashboard is not accurately displaying my request quota usage.
Due to severe rate limiting, I resorted to using DataStores for storing weekly and monthly user data—data I intended for MemoryStore leaderboards. Now, when players leave the game, I check if their stats exceed those of the 100th person on each leaderboard. If they do, I save this data to the MemoryStores; otherwise, I ignore it. According to the observability dashboard, my memory storage usage is under 5%. My ideal use case and what I did from the start was update leaderboard Memory Stores each time a user’s stat updated in-game, but this turned out to be a cause of lots of the rate limiting I was getting even though it was infrequent, so I had to remove it. It would be nice to have the user’s stats update on the leaderboard whilst they’re in game so they don’t have to leave just for them to save, but this simply isn’t feasible with the current rate limits.
Feel free to ask for my code or more details, and I can provide further information via DM to help investigate this issue. I could also try and reproduce this issue by adding in more frequent updates of leaderboards in my game (not just when a player leaves) and let you know if it incorrectly rate limits me again, and how this correlates to the stats shown on the observability dashboard.
Also, could you explain the difference between these two errors?
Looking back on my data from the Creator Hub I can see that if it’s correct, I wasn’t even close to hitting my specified quota. The section without the blue at all is when I begun using my API to retrieve and cache MemoryStore data on the server.
If additional information is of any help, I’m getting RequestThrottled errors in a completely fresh game that has barely any users. First time using the MemoryStores, so I thought I did something wrong, but there’s no way I would be able to hit request limits and stats don’t show anything abnormal.
To add more context, my game (ID 5791359229) is a single player experience and I do request GetRange (but only with 5 entry limit) as soon as player joins. I also do multiple MemoryStoreService calls in a single function in case that has a potential to be an issue.
Been starting to have these errors in studio as well. I know for a fact I am not hitting the limit because it literally happens the first time I invoke :GetAsync on a SortedMap in studio play mode in some cases. This only started happening recently.
I should have a limit of 1100 in studio play mode (1000 + 100 * [number of concurrent users]) yet it throttles after just a few API calls, sometimes even the first time. I don’t have any other scripts that use MSS either, so there’s nothing that should cause it to throttle.
As you can see I am barely scratching the surface of my API request quota:
A change has been made that has absolutely broken memory stores, not making the rate limits lighter, but instead more strict, as those above have reported. Please look into this. @Burgundy2014
My previous leaderboard system used MemoryStoreService, but due to recent CCU drop in my game i basically lost access to use the service, mine memory usage quota has reached a critical threshold because of Monthly and Weekly leaderboard at the game recent peak CCU, so i had to switch the whole system to the DataStoreService.
I don’t think that making leaderboards relying on MemoryStoreService is a good idea now, because As Abcreator mentioned, the memory size quota uses an 8 day trace-back period. Your memory quota does not instantly change when players leave. The quota is always at least 64KB + 1KB ⨉ [number of users] is not enough, i think increasing traceback days would be a better option (up to 14 days) as it would allow the stored values inside SortedMap to expire before removing the memory usage quota from an experience and not breaking it.
But for the Leaderboards that uses MemoryStoreService you have to manually clean the data stored inside it which is meaningless given that it allows to hold values inside of it up to 45 days.
same! I recently got around 300 errors in some minutes because my Memory store hit the max quota of requests, but once I Checked I discovered I only used 0.43% of the max memory usage… I would specify that my game as almost 0 ccus all the time, that may be the reason of the error.
