Strange DataStore CURL error causing DataStore service to fail most, if not all, of the time across many games


#21

Plus one.


#22

Checked statistics for phantom forces, and failure rate for datastores in the last 6 hours are 3%, which is up from around 0.14% last time i checked. Definitely an issue

Edit:
Last 6 hours: 3%
Last hour: 4.2% failure rate
Last 30 min: 4.4% failure rate


#23

This is occurring to myself and @loterman23, through studio - separate places.

Blue == GetAsync
Red == Timeout
image


#24

Affects both the release version and the private test version of my game (even just in studio) so I don’t think it has to do with intentional throttling. Has been going on for at least 3 hours. I figured it was just caused by heavy summer traffic for Roblox in general at first. For the past 10 minutes or so I haven’t been able to load stuff from DataStores in Studio at all.


#25

I am consistently getting this issue. This is happening in multiple games that I’ve checked.


#26

Possibly make this Roblox Critical? @RBX_Lua


#27

I agree with Dapale here on this. It’s definitely fitting for critical.


#28

Now’s the time to make your game work with DataStores being down if they aren’t already so we know how to get around it :eyes:


#29

I mean, in my case, if players’ data hasn’t loaded, they aren’t going to want to play anyway. I imagine that’s the same in most cases.


#30

Have fun paying lots of money a month for strong reliable databases, heh. I might start setting up a system for phantom forces to auto upload data if saves fail, then on rejoin, overwrite. Especially with datastores failing hard right now…


#31

I mean writing code around it so that your game doesn’t break if its DataStore reliant to start the game. Also can’t seem to get my DataStores to fail, in game and in studio test.


#32

Removed ROBLOXCRITICAL tag.

@Dapale @Xan_TheDragon @RBX_Lua
Please keep in mind that ROBLOXCRITICAL is not subjective, a bug report needs to meet criteria to qualify as ROBLOXCRITICAL: https://devforum.roblox.com/t/please-read-before-posting-steps-to-report-a-bug/24388

(Note: “Data loss” does not include data losses from bad datastore code. Datastore failures are to be expected from time to time and code needs to handle these temporary failures gracefully. I just checked my own games and the error rates are far from 100%, so while very inconvenient, it’s not an insurmountable issue.)

The issue has been escalated.


#33

Thank you for the report. Our engineering team is actively investigating this issue and we’ll follow up soon with more information.


Receiving an error with DataStore. Not sure how to fix
#34

Ohhhhh, this is why I’m getting tons of messages about lost data.


#35

So, if something like this DOES happen and we need to notify easier, what do we do?


#36

You file a bug report without a ROBLOXCRITICAL tag.


#37

And we should assume that’s going to be seen instantly?


#38

According to my data there is large-scale data store failure and players being forcibly removed from the game.

The best datastore code in the world won’t fix that


#39

Operations has internal status trackers that will catch these issues faster than users report on them, so the bug report is really just additional comments on the issue.


#40

How many of those are after a final retry?