Strange DataStore CURL error causing DataStore service to fail most, if not all, of the time across many games


#41

All of them? if I understand correctly, re-tries are handled “under the hood” and if an error happens then that is the final status.

edit: to clarify, these are datastore load errors, not save errors.


#42

Auto-retry isn’t enabled yet. Right now you should be retrying requests manually. If your game handles retries gracefully, partial outages like these shouldn’t have much impact aside from slightly longer request times as they’re retried. This includes both loading/saving data.


#43

To add to my post above
The statistics I have are Per attempt in code to save.
I log each individual datastore SetAsync if its successful or failed, then log the errors as well.

Which within 30 minutes, had
87,630 successful
4,086 failures
(4.4%)

They attempt to save again and again until successful.


#44

Something interesting to note:

Here are all the biggest datastore errors and how many times they have occured in the last four months total not including today:

And Here is the error occurring today:

This is not the same error that usually accompanies high-traffic outages. This is an anomaly


#45

@BOF had some issues with this on twitter too, so this post is relevant to him as well


#46

The problem is that, at least in my limited experience, datastore errors are usually not “gracefully” distributed among servers. I don’t have concrete data but from anecdotal experience usually some servers are reasonably fine while others barely get any requests accepted.


#47

Does sort of seem to be the case, here are some fail rates (as a percentage) Per server, in the last hour or so. From this data, I have 3 servers with 100% fail rates. The first server on the list had 17 requests, all failed, 2nd and 3rd, had only 1. They’re probably new game servers that just started.

Edit: More data by total requests and fail rate
image


#48

Thank you for the data!


#49

Any fix yet? Had to shut my game down completely :confused: But I don’t notice it in anyone else’s games :confused: please help…


#50

Yeah, still is in the same state as earlier :frowning:


#51

Still experiencing issues. It’s not 100%, but it’s enough to keep me from working on my game (data won’t load).


#52

Yeah same :confused:


#53

I’m having this issue myself

Luckily I have a system in place that doesn’t save your data unless it has loaded in. The player’s data will be safe, but it will get annoying when you’re stuck on the loading screen.

Also I have a console warn whenever it fails to get data
aata

This loading screen won’t go away until your data loads

Edit: Finally it loaded, but it took ages of pcall retries
afa


Receiving an error with DataStore. Not sure how to fix
#54

Same here, can’t load DataStore data for my game I’m developing/testing on.


#55

Experiencing issues in both studio and online


#56

same,broke my game


#57

Thank you for waiting for a response. This issue has been resolved! :slight_smile:


#58

Any background on what went wrong?


#59

haha lol I think that data could not be saved one is mine


#60

Roblox is in PST right? So the engineering team stayed up until 11:30pm to fix this, thank you!