And we should assume that’s going to be seen instantly?
According to my data there is large-scale data store failure and players being forcibly removed from the game.
The best datastore code in the world won’t fix that
Operations has internal status trackers that will catch these issues faster than users report on them, so the bug report is really just additional comments on the issue.
How many of those are after a final retry?
All of them? if I understand correctly, re-tries are handled “under the hood” and if an error happens then that is the final status.
edit: to clarify, these are datastore load errors, not save errors.
Auto-retry isn’t enabled yet. Right now you should be retrying requests manually. If your game handles retries gracefully, partial outages like these shouldn’t have much impact aside from slightly longer request times as they’re retried. This includes both loading/saving data.
To add to my post above
The statistics I have are Per attempt in code to save.
I log each individual datastore SetAsync if its successful or failed, then log the errors as well.
Which within 30 minutes, had
87,630 successful
4,086 failures
(4.4%)
They attempt to save again and again until successful.
Something interesting to note:
Here are all the biggest datastore errors and how many times they have occured in the last four months total not including today:
And Here is the error occurring today:
This is not the same error that usually accompanies high-traffic outages. This is an anomaly
The problem is that, at least in my limited experience, datastore errors are usually not “gracefully” distributed among servers. I don’t have concrete data but from anecdotal experience usually some servers are reasonably fine while others barely get any requests accepted.
Does sort of seem to be the case, here are some fail rates (as a percentage) Per server, in the last hour or so. From this data, I have 3 servers with 100% fail rates. The first server on the list had 17 requests, all failed, 2nd and 3rd, had only 1. They’re probably new game servers that just started.
Edit: More data by total requests and fail rate
Thank you for the data!
Any fix yet? Had to shut my game down completely But I don’t notice it in anyone else’s games please help…
Yeah, still is in the same state as earlier
Still experiencing issues. It’s not 100%, but it’s enough to keep me from working on my game (data won’t load).
Yeah same
I’m having this issue myself
Luckily I have a system in place that doesn’t save your data unless it has loaded in. The player’s data will be safe, but it will get annoying when you’re stuck on the loading screen.
Also I have a console warn whenever it fails to get data
This loading screen won’t go away until your data loads
Edit: Finally it loaded, but it took ages of pcall retries
Same here, can’t load DataStore data for my game I’m developing/testing on.
Experiencing issues in both studio and online
same,broke my game