Memory Stores Service Quota [Update]

dragonknightflies · November 23, 2022, 6:16pm

Hi Creators,

To make the Memory Stores service more scalable and fair for everyone, we are updating the throttle limit for the service. The goal is to ensure no single experience overloads the whole system and thus impacts everyone else. The quota will start taking effect on January 9, 2023.

API Request Limit

We are introducing a new concept called Request Unit. The total request quota for an experience is still “1000 + 100 x [num of concurrent users]” request units per minute. Most APIs will only consume 1 unit. The exceptions are GetRangeAsync() for sorted maps, and ReadAsync() for queues. Both APIs will consume units based on the number of items returned. For example, if GetRangeAsync() returns 10 items, 10 request units will be counted towards the total quota.

Any single sorted map or queue is also limited to 100,000 request units per minute.

Per Data Structure Limit

To support large-scale experiences while protecting the whole system, we are adding size and item count limits for a single sorted map or queue:

Maximum number of items: 1,000,000
Maximum total size (including keys for sorted map): 100MB

If your experience needs to store data that surpasses this limit, we recommend sharding them, e.g. through key prefix, into multiple data structures. This is generally a good practice to scale your systems.

We appreciate your interest and support of the service, and believe changes will better serve the community in the long term. To learn more about the latest quota and how to use the service, please visit our official documentation.

Cheers,
The Roblox Creator Services Team

system · November 23, 2022, 6:26pm

This topic was automatically opened after 10 minutes.

ChipioIndustries · November 23, 2022, 6:27pm

When will we have an API to check our game’s memory store request budget? This is effectively impossible to track manually.

filiptibell · November 23, 2022, 6:58pm

This is going to make using sorted maps for leaderboards even more of a hassle we already need to use strange workarounds with prefixes and insert all the data into keys since sorted maps cant be sorted by values. Now the maps also need to be sharded and fetched separately to check top player scores, to not hit this new limit that there’s no way to check if we are actually reaching or not…

I’m not trying to rant but at this point it is so much more straight forward to just spin up a small Redis instance and use their leaderboard-like zset commands, and probably cheaper too, given that development time is a pretty high real-world cost and the amount of complexity with MemoryStore sorted maps is so high. Maybe MemoryStoreService just needs some kind of dedicated leaderboard-like datastructure similar to Redis zsets or ordered data stores.

sleitnick · November 23, 2022, 7:17pm

DataStoreService lets you query the current budget. I don’t believe MemoryStoreService allows this yet. This makes any budget (especially changes) hard to account for, since they have to be carefully hardcoded. Would we ever get a similar method for memory stores to query the current budget?

dragonknightflies · November 23, 2022, 7:23pm

Hi, just want to understand your use case, if we offer a budget query API, how would you use it in your code?

PeZsmistic · November 23, 2022, 7:24pm

+1, I cannot make my systems reactively adjust polling/publishing rate according to budgets without API for this. I currently would have to calculate out the maximum number of servers/players my server browser could support for example, and remember to change related rate values if my game ever starts to push it.

Especially now after this change.

sleitnick · November 23, 2022, 7:28pm

e.g. for matchmaking, I could dynamically throttle my machmaking intervals to ensure I keep within the budget, especially if reading a bunch of items with GetRangeAsync takes a big chunk of the budget.

DataBrain · November 23, 2022, 8:23pm

In my own experience’s code, I always throttle any and all DataStoreService requests until a minimum budget is reached. This introduces a natural throttle that ensures the right requests are prioritized, and no requests are put on queue.

For example, for saving player data, I would do this (such that a request is never made while the budget is zero):

while
	DataStoreService:GetRequestBudgetForRequestType(
		Enum.DataStoreRequestType.SetIncrementAsync
	) < 1
do
	task.wait(1)
end

playerDataStore:SetAsync(playerKey, playerData)

And for any other DataStore requests, I make sure there is always enough headroom for saving player data:

while
	DataStoreService:GetRequestBudgetForRequestType(
		Enum.DataStoreRequestType.SetIncrementAsync
	) < (game.Players.MaxPlayers + 1)
do
	task.wait(1)
end

nonPrioritizedStore:SetAsync(key, value)

This allows saving player data to take priority over all other types of data store requests, in order to prevent data loss / slow data saving.

So, when querying global limits to memory store requests I would simply do the same thing. This would be VERY useful to have for all experiences I work on.

Request limit APIs allows the code to naturally throttle and never overload the system, OR to prioritize esential game systems while the system is overloaded.

Rootie_DaHoodie · November 23, 2022, 9:06pm

Are there any plans to up the limit for the open cloud API’s datastore? We’re not able to use it in our game Fashion Famous as we’re hitting the request limit way to quickly.

The * 300 reqs/min/universe is no where near enough considering we see 10k CCU’s.

dragonknightflies · November 23, 2022, 11:27pm

Thanks for sharing the code! It’s very helpful to understand your expected use case. For memory stores, the budget many change very quickly given the # of server that access it. So do you think such information will be as useful as data stores budget?

TheH0meLands · November 23, 2022, 11:40pm

oh wow, thats cool. Is there any plans to add a way to track this data? It’s impossible to manually do so.

Abcreator · November 23, 2022, 11:43pm

Datastore limits use the same sort of quota system (based on the number of active players in servers) and still has an api to query budgeting. So MemoryStoreService having this API would be just as hugely useful.

imNiceBox · November 24, 2022, 7:22am

I made some calculations and I’m honestly concerned about the change.

In my game there’s a cross-server matchmaking system which puts the players into a sorted map (more comfortable implementation than Queues).

The starter servers are 8-player each with 3 slots reserved for friends.

Each 2 seconds each server makes GetRangeAsync request which returns all the joinable matches.

Now, according to what I’ve said above:
(matches_in_map * (60 / 2) + 60@) * lobby_servers >= min(1000 + 100 * players, 100000)

60@ are additional requests related to the matchmaker.

Now, using estimates and data collected so far by the testers I am able to replace the variables quite accurately using CCU variable. It would be:
(ccu / 35) * 30 + 60) * (ccu / 12) >= min(1000 + 100 * ccu, 100000)

Which gives ~ [1150, +inf] (this already uses 100k requests quota), meaning that 1150 would be the maximum concurrent user count in my game after this change. Actually, probably a little more because I used really safe estimates.
While I can only hope for my game to reach that ccu, it appears to be a huge problem for huge games.

Of course, I could increase this number by for example:

increasing the delay between expensive requests
increasing maximum player count per server
adjusting other matchmaker settings

But it causes more inconveniences and still, will eventually exceed the new quota.

Therefore, my suggestions to change the current way of constraining the requests would include:

force a delay between expensive requests (0.5 second for example), like in DS.SetAsync, as calling these functions on a single map/queue in a single server more frequently is a terrible practice anyway
change the linear formula of calculating units of expensive requests to an sqrt function so for 200 results the unit count won’t exceed 50.

I hope there will be changes to the current quotas as they are harmful, especially for matchmaking systems.

sndiax · November 24, 2022, 7:28am

What the current request per minute right now?

JuniorMikezz · November 24, 2022, 12:01pm

Does this mean that DataStoreService 's

will be fixed? i.e making saving more effective and secure ensuring that no data is loss because too much data was saved at the same time?

BackspaceRGB · November 24, 2022, 3:26pm

What does this have to do with the original post…?

Also, pretty sure that data doesnt get lost because of that, it just takes longer to save

Bitsfps · November 24, 2022, 5:15pm

Why are you making constant, expensive requests every 2 seconds? isn’t this REALLY wasteful for no practical changes at all?

DataBrain · November 25, 2022, 2:21am

Even with a heuristic value that isn’t always up to date would be useful. The main concern is avoiding request throttling and/or making sure core systems always remain stable.

Hollistic · November 26, 2022, 4:41pm

Finally. Big games won’t crash other games anymore. Was incredibly inconvenient and unfair for everyone. I am tired of my memory stores breaking because some enormous game was crashing the system.