Datastore Issue: " 502: API Services rejected request with error. HTTP 503 (Service Unavailable) "

[1billybob1]

Hello,

ISSUE/DESCRIPTION:

Beginning approximately one day - two days ago, I have been getting reports of broken servers from players of my game (Tornado Alley Ultimate - Roblox).

After investigating these broken servers myself, a specific error has been found prevalent in every broken server: “502: API Services rejected request with error. HTTP 503 (Service Unavailable)”.

I believe this error is the result of DataStore request queue being filled up. However, I’ve made no changes to the game that could have stemmed this issue, it just began breaking servers within the last day or so.

For five months, my game has been working fine, no throttling warnings or request queue issues, up until about a day - two days ago. This is when I began getting reports of broken servers.

I have two DataStores: An OrderedDataStore for Global Leaderboards and I am also utilizing DataStore2.

No changes to these DataStores or how they operate were made.

Initially, I believed this may have been an issue with my implementation of DataStore2, however other developers are experiencing this HTTP 503 (Service Unavailable) issue without the use of DataStore2 (such as developer @Floo_d) . I also thoroughly read DS2 documentation and ensured I am correctly using it.

EXAMPLE:

I’d like to explain what usually occurred within a broken server by reading the server console logs:

• Game begins warning DataStore request queue is filling up.
• “502: API Services rejected request with error. HTTP 503 (Service Unavailable)” error prints
• DataStores no longer update

Here is an example of the error being thrown (Note: the mouse is covering the 502- apologies*) [This is not DataStore2]

Another example: [This is DataStore2]

Once again, I’d like to state this was never an issue (nor were these warns/errors prevalent) up until about a day - two days ago (this is the same time other developers such as @Floo_d and @static2240) reported these issues on a different thread. My game was working perfectly for five months.

Potentially related, when this error started becoming prevalent in my game, it happened to be around the time players were reporting an issue with ROBLOX (June 8th at around 6:00 PM).

Ignore the handwriting, but as you can see, as I began to gather reports of broken servers in my game, the website “downdetector” was reporting a spike in issues with ROBLOX.

Again, I am not sure if this is related to that (or if it’s related to the ROBLOX maintenance that occurred last night on June 9th), but it started occurring the same day around the same time. However, the difference is that this issue has not gone away.

As one of the developers stated with this issue: “Seems that the DataStore queue overfills too fast”.

REPRODUCTION

Unfortunately, I have not been able to reproduce this issue by myself. However, it became so prevalent within my game servers (holds 22 players to a server) that I have had to temporarily close the game until this issue is solved. If I’m able to reproduce in some fashion, I will post what I find here.

With 22 players to a server, after a duration, request queue fill warnings begin showing up. Shortly after, the 503 error shows up.

And lastly, to sum it up:

• I do not believe this is a DataStore2 issue as other developers have been reporting this issue without using DataStore2.
• I do have API Services turned on, but I am still getting this issue
• The issue started occurring approximately 1 - 2 days ago (I believe on June 8th).

It does seem like DataStore requests are filling up unreasonably faster than before. This has never been an issue in my game before.

If anyone else has this issue, please feel free to post your experience with it. Or, if you know how to reproduce it, please share it here.

[daanemark11]

I too have been having this issue.

I have been using datastore2 and maybe twice a week this error happens and someones data isnt stored
https://gyazo.com/7eb3254be5d78e6fc6ee750c6c6edf65

This is the code where the error failed

function OrderedBackups:Get()
    return Promise.async(function(resolve)
                --ERROR HAPPENED WITH THIS LINE VVVV
        resolve(self.orderedDataStore:GetSortedAsync(false, 1):GetCurrentPage()[1])
    end):andThen(function(mostRecentKeyPage)
        if mostRecentKeyPage then
            local recentKey = mostRecentKeyPage.value
            self.dataStore2:Debug("most recent key", mostRecentKeyPage)
            self.mostRecentKey = recentKey

            return Promise.async(function(resolve)
                resolve(self.dataStore:GetAsync(recentKey))
            end)
        else
            self.dataStore2:Debug("no recent key")
            return nil
        end
    end)
end

So as you can tell this is an error that happens to me directly in datastore2.
I sadly don’t know the fix either but I am doing my best to figure it out.

[1billybob1]

I have decided to reopen my game in hopes to catch screenshots of this issue in action in the console logs. Unfortunately this issue is completely breaking my game, so I really hope there will be some fix soon. Anything that I catch there I will send here.

[Scrizonn]

Having the issue here and also using datastore2. This hasn’t happened before and I haven’t made any adjustments to the games datastore or any changes recently to the game.

[1billybob1]

Here is an example of how the issue comes about in my servers:

[Husslematic (Matic)]

This has been occurring in my game as well:

[therealbiker]

Thank you for your reports.

This has been corroborated by some of our internal metrics, we are working on a resolution.

[Nahr_Nahrstein (Nahr)]

Tradelands has been seeing this issue occasionally for a few weeks now, with a large spike on the 8th and 9th.

A few variations of the error:

(The first has happened about 500x as often as the others, but the others are the ones that’ve been happening for weeks):

• 502: API Services rejected request with error. HTTP 503 (Service Unavailable)
• 502: API Services rejected request with error. HTTP unknown error (HttpError: curl:52, Server returned nothing (no headers, no data)) |
• 502: API Services rejected request with error. An exception was thrown when attempting to send the request. Url: (Post) http://gamepersistence.api.roblox.com/v3/Set?apiKey=********
• 502: API Services rejected request with error. HTTP unknown error (HttpError: SslConnectFail)

[therealbiker]

This specific problem has now been resolved:

The cause was an internal misconfiguration that caused a small number of requests rejected, just low enough to avoid automatic detection. Thank you for reporting it!

As far as other errors go, some number of errors is inevitable because of the nature of the Internet. There’s always a possibility that some connections get dropped and some requests don’t complete, that’s why defensive coding is important. We do try to bring all errors that are in our control (like this one) to zero though.

[1billybob1]

Thank you!!! Zero issue reports coming from players now!

[Jelotto]

I’m not sure if this is related but when I shutdown my game, players are teleported to another game then back. When everyone joins back, the datastore begins to queue (Im using datastore2 and Ordered for leaderboards). I have seen an error with “promise”. I detect if their data is loaded if not I force rejoin them and their data is either back or reduced (Some say reverted back to their old stats). This could also be an issue on how I’m loading their stats on their UI when the join… Any suggestions?

[MXKhronos (Nate)]

This appears to be happening to my Studio plugins, I have multiple plugins that’s used to access save data and I have made sure that the place is published.