Over the past ~3-4 days we’ve been seeing a significant increase in errors in our usage of MemoryStoreService. This started right around ~4pm PST on October 28th, 2025.
Full message sent privately.
A private message is associated with this bug report
Hi @asimo3089, could you tell us your experience ID and your Memory Stores usage pattern (what data structure, what do you typically store)?
Jailbreak is now very down, seeing 10% our typical CCU.
Experience ID is: 245662005
Tagging in @badcc for more info.
We’ve had memory store issues before. Could be related? Degraded MemoryStoreService (SortedMap) for past ~16 hours (affecting entire player base)
We’re seeing an significant CCU drop since this morning, and an elevated number of errors regarding MemoryStores, with the backend returning (Code 500: InternalServiceError)
Thanks for the report, we are working on resolving this issue right now.
We added some info in private message, but I will make public here:
We’ve been seeing a significant increase in errors in our usage of MemoryStoreService → HashMap → UpdateAsync. In-game these errors are reported as RequestThrottled, and in Creator Hub they are reported as DataUpdateConflict.
We use these HashMap’s for session locking to prevent people from loading their player data into multiple games, and have done this same approach for the past few years now. The problem is when the memory stores fail, it kicks all of our players. So our concurrent player graph has been incredibly spikey over past ~4 days, and we’re getting tons of reports from people.
Another weird thing is that despite the memory store hashmap being used across ALL GAME SERVERS, we’ll see ONE SINGLE entire game server go down at a time… a typical memory store crash I would think, and what we’ve seen in past, would see ALL game servers start failing. But, I don’t know how the new partitioning stuff works. Maybe related.
In this Memory store monitoring page (picture attached) we see our “quota” start doing really weird things .. is this expected? Our usage seems WELL below the quota. And a second photo attached not of the quota but of just the DataUpdateConflict errors themselves starting at the same time, ~4pm PST.
Hey all. Wanted to give some additional context here.
There was an issue this morning in which nearly all HashMap operations were failing. This issue has been identified and fixed.
Separately, we are continuing to investigate abnormal behavior related to RequestThrottled, DataUpdateConflict, and quota lines, as you included in your report.
We’ve identified and fixed the originally reported issue. Thanks, as always, for your patience!
The fix was completed at 3:10pm PDT. If you switch your monitoring page to “Last 1 hour” with “Minutes” interval (direct link), you should see things are back to normal.
Please let us know if you see any lingering problems!
Wow! It sure is fixed!
This has affected our player base very much over past few days, .. servers were randomly crashing frequently, our player counts spiking down every. We spent many hours and late nights trying to diagnose, because our community had us thinking it might be exploiters crashing servers. But after pulling our hair out long enough, signs started to point to it maybe not being something in our control.
So please for our sanity.. is there any more info you can share on what sort of problem it was, or what solution was?? For example: why it was only one server at a time? We have wrestled with MemoryStoreService reliability a lot over the past few years, but depend on it.
Thank you very much for your help today and any additional information on your side you can satiate our curiosity with.
We have wrestled with MemoryStoreService reliability a lot over the past few years, but depend on it.
First, I want to say that our team is dedicated to continuously improving this reliability for you all. I can definitely appreciate wanting more info here.
The root cause was a bug in our quota determination.
As you pointed out, your quota line appeared sporadic. In rare conditions, MemoryStoreService determined your available quota to be either very high or very low (near-zero) on specific servers. This means that, for brief periods, specific servers believed you to be out of quota.
This affected a very small percentage of total requests, but (as you pointed out), it frustratingly affected all requests from a specific server when it happened.
Makes sense, thanks for that info!
Memorystores are out again 8 days later, and players cannot join games like Tower Defense (@OutOfBears) or Jailbreak.
Unsure if related to last week’s outage.
Our engineering teams are aware and investigating. Thanks for bringing this to our attention!
Our game Fishies is affected too
Yup! This is the case, this seems to be a reoccurring issue that happens every weekend when platform CCU spikes
Hey everybody, the issue should now once again be resolved. Thank you for bringing this to our attention!
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
