DataStores are having large issues on certain servers


#1

As of recently, I have been tracking datastore errors, and I am going to keep updating this post with information of what the errors are, etc.

From information i’ve gathered, it appears this particular spike in errors only applied to 3 game servers, which means all players in that server most likely had data loss.

How often do they occur?
From what I’ve had so far for stats, it seems to peak in errors every few days

Where does it happen?
Phantom forces https://www.roblox.com/games/292439477/Phantom-Forces

We recently added a 6 second wait between saves to ensure its not us going over the limit (plus other measures)

Would a screenshot or video help describe it to someone?
Here is a custom statistics page I created, each individual error or success is Per datastore save attempt (SetAsync) (also ignore the drop in saves at the end, thats just the way it gets the data)

The majority of the errors contain this error message:
502: API Services rejected request with error: HTTP 0 (HTTP 403 (HTTP/1.1 403 Forbidden))

Edit: after like 16 minutes, it fixed itself
image

I’d like to add more information that may be helpful:
I grouped errors by server ID, and it appears to be some servers only getting save issues.
server_id is a unique increment number for each JobID, and total_errors is a count of save errors alone.
image


#2

It would also help if you could time stamp the chart itself so we can have a frame of reference for when each of the datapoints are sampled.


#3

Very first datastore error for the server was at (Unix timestamp) 1530563368
The chart is based on when we look at it, i got to add formatted timestamps, just easier to see minutes ago.