Internal Services Are Down; Players Are Already Losing Data & Robux

I’ve been closely monitoring my game for the past few minutes, and everything appears to be back up and running as intended, except for Kohl’s Admin:
image

It has caused many players lost their savedata.
Their lv back to 1.

Depends on the game, some such as Natural Disaster Survival seem to be fine. I’ve also experienced this.

1 Like

FIltering seems to be fine, Marketplace, Datastores are down

Accessing MarketplaceService functions returns a “InternalServiceError” and Datastores return 503

how do you, as a developer - rollback data, or merge 2 datasets?
im using datastore2 module, and cant even comprehend how to handle this type of error.

games using httpservice to save data :dab:

I’m not one of them : (

1 Like

This is fixed now, I see it’s resolved or as I read on the status roblox thing.

Luckily for most European players this only happened at night since now services have returned to normal.

1 Like

yes,it’s lucky. it’s peak period of Sunday at asian.

Losing data during/after service outages has nothing to do with luck or anything Roblox can resolve. The developer needs to properly interact with datastores and not overwrite your data with default data when a previous fetch call failed. This is 100% on the developer writing faulty code, so don’t expect Roblox to help out here, ask the developer of the game you are playing to refund or rollback your data.

6 Likes

Nope, I am european and it wasn’t night when it was happening. (At least for me)

This issue is still persistent although not as frequently. I am still getting reports of people not loading into the game as well as Internal server errors.

2 Likes

Can confirm this is the same error being thrown in my experience, still occurring as of 11:15 AM EST.

Causing some servers to break entirely.

1 Like

Issue is still prevalent as of 11:41 AM EST. To the point in which I’ve had to temporarily close my game because of the HTTP 500 (Internal Server Error) message appearing in every server, followed by the server breaking entirely.

Edit 12:34 PM EST. this is my error I’m receiving:

Would really appreciate if an engineer can look into this. My game has had to be down for over an hour now and I’m unsure what to do.

There is definitely still an issue here, the issue has not been resolved

Seems like a steady amount of players are continuing to report issues as well on the down detector:

EDIT 2: 2:13 PM EST

I can confirm that I am not seeing these issues any longer. I was able to reopen my game, and not a single report of this issue. Perhaps it just took a bit for the servers to be fully operational again,

As such, my problem (at least in my case) is resolved.

1 Like

Our experiences is also having the same problems. However, one specific game have the servers entirely broken.
When joining, it creates a server, put your user in it then immediately disconnects you out, attempting to “Reconnect” triggers an ID=17 code.

This caused the game to clone multiple servers with the same single player.

image

Yet, all other experiences work flawlessly other than the Roblox’s datastores being unaccessible at the moment.

1 Like

I’m guessing a lot of games on the platform do this too for their data systems which is really bad…

ProfileService does that.

1 Like

Just wanted to mention, in case it helps anyone. We had a bunch of reports in the last few hours of people saying that they couldn’t get into our game as they were stuck on our portal’s loading screen.

When I investigated, I found a couple of servers that had running since the datastore outage, that still weren’t able to talk to the datastore. They weren’t raising errors just appeared to be hanging on datastore requests.

We restarted the whole place and all seems to be okay now…I hope :frowning:


Edit: okay, now I know what happen. One of our core modules makes a call to the datastore while it’s loading. So when the datastore breaks, anything that requires that module will also break.

And everything requires that module. Whoops!

Don’t be like us :slight_smile:

Just a tip in this regard, I highly recommend making your require graph “static”: That is, that your Modules never do anything other than define functions and classes when required, with things only happening once the requiring script uses those functions / classes in some way.

Otherwise you’re inviting a lot of issues, like this, and also accidentally changing the behavior of your place just because you changed the order of require calls in an entirely unrelated part of your place.

Thanks!

We’re trying to move to purely stateless modules. Especially in prep for Parallel Lua. We just have a little bit of technical debt to work through :slightly_smiling_face: