"502: API Services rejected" DataStore error becoming increasingly common

Bug Reports Engine Bugs
#1

We are regularly seeing the following datastore errors:

502: API Services rejected request with error. HTTP 0 (HTTP 403 (HTTP 403 (Forbidden)))
502: API Services rejected request with error. HTTP 0 (HTTP 429 (HTTP 429 (Too Many Requests))).

These have been specifically noticeable this summer, and appear to be entirely related to scale. Two days ago for instance, there was a spike in these over the span of about 6 hours. I have spoken with other top games and they are also seeing the same thing. We have had to develop our own internal tracking and notification system in order to know when these errors occur.

Here is a graph errors over the last two weeks.

image
image2096Ă—1396 218 KB

It seems odd to me that more popular games don’t have greater datastore budgets reserved, and that it’s only based on server player count. Is this something that will ever be considered? It seems to me that these errors are occur when requests are being rate limited - that should really never be the case if we are always respecting the DataStore limits.

1 Like
#3

@Ice7 - Thank you for bringing this to our attention. Can you share details on which API is returning this error?

2 Likes
#4

Hi! So this appears to be happening for :SetAsync() calls with regular DataStores.

1 Like
#5

Thank you, we will investigate and follow up

1 Like
#6

We are still investigating, will keep this thread posted.

1 Like
#7

We saw another large spike of “502: API Services rejected request with error. HTTP 0 (HTTP 403 (HTTP 403 (Forbidden)))” in the hundreds today and yesterday - I managed to go in game while this was happening and my data didn’t save. Once the fails cooled down, it begun saving again.

image
image794Ă—791 56.6 KB

#8

@Ice7 - Thanks for the follow up, we have continued to investigate and are closer. Will keep this thread updated with new developments.

2 Likes