Since Thursday morning I have tracked 3,887 cases of :SetAsync() calls getting stuck - not returning anything. These correspond to 90 unique cases. By unique cases, I am referring to 90 different user ids. As 90 may not seem like many, I am very concerned as to why this is happening for these particular user ids.
How do we know they are getting stuck?
I am setting a flag after our datastore retry function completes. The function accepts a datastore function, such as SetAsync, or GetAsync, and that datastore function is called inside a pcall. It only retries 5 times, or until there is a successful response. If the flag has not been set after all datastore retries complete, then I am logging this event to our external backend. In other words, the :SetAsync() is getting stuck with no error messages.
Another reason I know this is happening is because we are unsetting the player from the session data cache as soon as our datastore function completes. These players are never getting unset from the data cache, meaning that even after they leave, our autosaving function continuously attempts to save their data. I can get around this by checking to see if the player is actually in the server before it tries to save their data, but it won’t change the fact that the calls to :SetAsync() are getting stuck,
What I can gather from all of this is that its happening several times for the same players, so the datastore servers are timing out for these keys (user ids).
The game is Royale High, so the total number of :SetAsync() calls overall in all servers collectively is several thousands per minute. Of these millions of :SetAsync() calls since Thursday, there are only 3,887 cases where they are getting stuck - so it’s an issue that’s only noticeable at this scale.