Degraded MemoryStoreService (SortedMap) for past ~16 hours (affecting entire player base)

We depend on SortedMap MemoryStoreService uptime when saving player data in Jailbreak. Around ~16 hours ago we started seeing tens of thousands of players affected by some service degradation that has not subsided. Here is a graph of the errors going from 0 to tens of thousands.

We have not updated our game in ~4 days at ALL, and the last time we touched this data store code was well over a month ago. We are well within memory store budgets, etc., utilizing on average just ~7% of our budget, so it’s nothing there.

The other info is that we aren’t getting an explicit error from MemoryStoreService… it’s just that it’s taking >30 seconds to return (when it should take less than a second), and so we’re timing out the request ourselves.

10 Likes

Is this because regular DataStores are not reliable enough?

We’ve found even UpdateAsync isn’t truly atomic and will return bad (old) data, so I have seen people using MessagingService or MemoryStores to try and patch this.

4 Likes

We use it to create a lock on player data so that stale data cannot be loaded in another session, or so that we can inch closer to having transactional security.

3 Likes

This issue has been resolved via an internal support ticket. Thank you to everyone involved

4 Likes