Analytics displays the game as being orders of magnitude below request unit quota, also far below the limits for an individual key, and yet the game is regularly getting throttled during player spikes preventing matchmaking from functioning as expected.
Either analytics are showing the data or the quota incorrectly, there is a bug with the throttling, or there is some undocumented limit to MemoryStore requests I am unaware of.
Attached are graphs showing the increase in MemoryStore errors during player spikes, despite not being close to quota according to analytics.
The ID of the specific experience this is occurring for (though potentially others also) is 3929033413.
The only relevant documented MemoryStore limits are:
100,000 requests per minute universe wide for a single key, which I can’t be hitting as my total requests per minute rarely exceeds 25,000.
1000 + 100 * [number of concurrent users] request units per minute, which per analytics I am orders of magnitude below.
I can provide more details if needed.
Thanks
4 Likes
Thanks @LordHenryVonHenry. I have my team take a look.
3 Likes
Is there any updates? I am using memory store for session lock in my game, and some of players are getting stuck in the loading screen due to the last server throttled the memory store requests
1 Like
@ivorycastle Since you responded here and seem to work in the Memory Stores area, could we get an update on this? I’m really unsure why this is happening. Is there a per-server throttle for MemoryStoreService SortedMaps that isn’t documented?
@Burgundy2014 It’s been nearly 3 months since you said you were going to look into this. Can we get an update or will we continue to be ignored?
1 Like
@bvetterdays @Intertruth @LordHenryVonHenry
Sorry about the delay. We have in fact been working on this. There are have been a few Memory Store quota limits tickets and issue, so we have been trying to disambiguate the problems from each other.
I want to clarify the problem that this post is for.
My understanding is that it is the experience-wide API request limit that is being hit:
1000 + 100 * [number of concurrent users] request units per minute
Is this only happening for small games with few concurrent users? Can you let me know your experiences user ID and average concurrent users?
If the above is the issue, we believe we understand the cause and are exploring options to fix it. The issue is finding the most accurate way to share a global quota in a distributed system. We though our original system was effective and have already tweaked it to solve an earlier issue, but we may need a larger scale redesign.
Thank you for your response—I appreciate your efforts. My experience ID is 6321891709.
Reflecting on this, it seems possible. In my scenario, I was invoking :GetRangeAsync on 9 different SortedMaps in each server every 60 seconds, primarily to refresh leaderboards every minute. Initially when the problem begun I changed the refresh time to 2 minutes, then 3, then 4 and 5, and this still did not resolve the issue. As problems persisted despite my efforts to resolve them, I shifted to using my own external API. The API retrieves SortedMap data from https://apis.roblox.com/cloud/v2/universes/UNIVERSEID/memory-store/sorted-maps/leaderboard/items every 60 seconds, caches it, and then my game servers request it via HttpService.
However, this raises questions about the accuracy of the Memory Store observability dashboard. The issue is either:
- I am being rate-limited too frequently than the documented rate limits.
OR
- The rate limiting is accurate, but the MemoryStore observability dashboard is not accurately displaying my request quota usage.
Due to severe rate limiting, I resorted to using DataStores for storing weekly and monthly user data—data I intended for MemoryStore leaderboards. Now, when players leave the game, I check if their stats exceed those of the 100th person on each leaderboard. If they do, I save this data to the MemoryStores; otherwise, I ignore it. According to the observability dashboard, my memory storage usage is under 5%. My ideal use case and what I did from the start was update leaderboard Memory Stores each time a user’s stat updated in-game, but this turned out to be a cause of lots of the rate limiting I was getting even though it was infrequent, so I had to remove it. It would be nice to have the user’s stats update on the leaderboard whilst they’re in game so they don’t have to leave just for them to save, but this simply isn’t feasible with the current rate limits.
Feel free to ask for my code or more details, and I can provide further information via DM to help investigate this issue. I could also try and reproduce this issue by adding in more frequent updates of leaderboards in my game (not just when a player leaves) and let you know if it incorrectly rate limits me again, and how this correlates to the stats shown on the observability dashboard.
Also, could you explain the difference between these two errors?
Looking back on my data from the Creator Hub I can see that if it’s correct, I wasn’t even close to hitting my specified quota. The section without the blue at all is when I begun using my API to retrieve and cache MemoryStore data on the server.
