Sharp increase in DataStore failures every weekend

[berezaa (Bereza)]

Since late March, 5 out of 6 weekends have experienced a sharp increase in datastore failures:

If you zoom out a bit, you see this isn’t a recent problem:

Almost every weekend, usually around Friday night or Saturday morning, there is a sharp spike in DataStore failures.

At times up to 1% of all requests are rejected per day.

This is the most common error:

It has appeared over a thousand times on some days:

This is not the only error that occurs:

1% failure rate does not sound that bad… until you realize this failure rate is from an entire day and that these errors are usually concentrated into just a couple hours of peak player activity. Meaning that the effective error rate during these outages is likely much higher.

This consistently poor reliability during peak hours is unacceptable and should be addressed.

[Coeptus]

Can confirm. Receiving several reports of DataStores failing on weekends.

This is a serious reliability issue that affects a lot of players.

[UnderMyWheel (Wheely)]

This is definitely an issue that Roblox engineers need to address immediately. Player’s losing data in games will only give Roblox a bad reputation among developers and gamers. Game developers will also get blamed for scamming due to Roblox failing to keep the data store reliable at peak times.

I know Roblox is constantly releasing knew updates, but perhaps it is time they slow down or make changes because there are many issues that are constantly popping up from date store failures to new features such as game passes having many issues that should have been caught before release. Due to the most recent update, most of us can’t even use play solo anymore which is extremely problematic too. And for a platform as large as Roblox, this is unacceptable and needs to be immediately addressed.

[berezaa (Bereza)]

Ah… right on time

[Noble_Draconian]

Been having the same issue.

Yesterday (On Saturday) I was getting a lot of 502 throttle messages for my ordered datastore requests.
Problem is, I had plenty of available datastore requests. I checked the datastore budget tab in the developer console right after the throttle message displayed, and the tab said I had 32 available ordered datastore requests. That should not have been throttled.

Here’s the throttle message.

Even more oddly, the datastore request executed successfully right after the throttle message was displayed.
Notice how the request completes successfully right after the throttle message.

[JParty]

@berezaa I sent you a PM requesting details, let’s see if we can get some answers around this internally.

[EncodedLua (Juniper)]

Yea, I got a few reports of data loss in my game as well (around 200-300 people playing at once on the weekend) however the data saving system I use has a couple of safety nets to prevent data corruption or data loss so this might be the cause as I’ve seen some api errors in studio when I test.

[berezaa (Bereza)]

So far it looks like this weekend isn’t experiencing a major crash!

(faded color is last week, line is this week)

[Noble_Draconian]

Thought I would bring this up, for whatever reason writing to a key in an Ordered Datastore randomly fails to complete within <30 seconds.

19:40:13.088 - Not running script because past shutdown deadline

I have a BindToClose function that yields until any player data in the server is saved (in the event of a server being shut down, their data is written to ordered datastores). This works most of the time, but ordered datastores sometimes take 30+ seconds to execute a write query. I noticed it happening a lot on the weekends specifically.