Strange DataStore CURL error causing DataStore service to fail most, if not all, of the time across many games

And we should assume that’s going to be seen instantly?

According to my data there is large-scale data store failure and players being forcibly removed from the game.

The best datastore code in the world won’t fix that

9 Likes

Operations has internal status trackers that will catch these issues faster than users report on them, so the bug report is really just additional comments on the issue.

1 Like

How many of those are after a final retry?

All of them? if I understand correctly, re-tries are handled “under the hood” and if an error happens then that is the final status.

edit: to clarify, these are datastore load errors, not save errors.

1 Like

Auto-retry isn’t enabled yet. Right now you should be retrying requests manually. If your game handles retries gracefully, partial outages like these shouldn’t have much impact aside from slightly longer request times as they’re retried. This includes both loading/saving data.

2 Likes

To add to my post above
The statistics I have are Per attempt in code to save.
I log each individual datastore SetAsync if its successful or failed, then log the errors as well.

Which within 30 minutes, had
87,630 successful
4,086 failures
(4.4%)

They attempt to save again and again until successful.

2 Likes

Something interesting to note:

Here are all the biggest datastore errors and how many times they have occured in the last four months total not including today:

And Here is the error occurring today:

This is not the same error that usually accompanies high-traffic outages. This is an anomaly

4 Likes

@BOF had some issues with this on twitter too, so this post is relevant to him as well

1 Like

The problem is that, at least in my limited experience, datastore errors are usually not “gracefully” distributed among servers. I don’t have concrete data but from anecdotal experience usually some servers are reasonably fine while others barely get any requests accepted.

Does sort of seem to be the case, here are some fail rates (as a percentage) Per server, in the last hour or so. From this data, I have 3 servers with 100% fail rates. The first server on the list had 17 requests, all failed, 2nd and 3rd, had only 1. They’re probably new game servers that just started.

Edit: More data by total requests and fail rate
image

3 Likes

Thank you for the data!

Any fix yet? Had to shut my game down completely :confused: But I don’t notice it in anyone else’s games :confused: please help…

2 Likes

Yeah, still is in the same state as earlier :frowning:

Still experiencing issues. It’s not 100%, but it’s enough to keep me from working on my game (data won’t load).

1 Like

Yeah same :confused:

I’m having this issue myself

Luckily I have a system in place that doesn’t save your data unless it has loaded in. The player’s data will be safe, but it will get annoying when you’re stuck on the loading screen.

Also I have a console warn whenever it fails to get data
aata

This loading screen won’t go away until your data loads
aatag

Edit: Finally it loaded, but it took ages of pcall retries
afa

1 Like

Same here, can’t load DataStore data for my game I’m developing/testing on.

2 Likes

Experiencing issues in both studio and online

same,broke my game