My game Mini Empires has received no change to its teleporting logic, and was earlier this year able to sustain 7,000 CCU. However, lately the amount of teleport failures has increased exponentially according to my players and this graph:
This issue is critical to Mini Empires as it is a game that heavily relies upon its own custom matchmaking systems in order to bring players together. Some matches require all players to successfully connect in order to begin, but at the moment at least 1 player does not connect about 70% of the time.
I have tried shutting down my game for an hour in an attempt to reset the limit counter on Roblox’s end, but unfortunately to no avail.
The game had about 100 CCU at the time when it was made public again, however the errors indicate that it is still exceeding 1800 API requests/minute - which is highly unlikely as only 20 of these players were matchmaking at the same time.
Hi @Avallachi, thanks for the report. With the teleport issue we solved last week due to the misconfiguration, do you still see that you need to retry a lot for teleport? If you do then we probably should figure out that first, what’s the error you’re experiencing now with teleport request?
Also, I wonder if there’s any change to your experience. Not just the coding changes, do you notice changes like the increased number of players that your experience tried to teleport all together?
As a temporary workaround, can you throttle the retries? like wait some time before next retry if hitting the limit.
As other developers stated in the original bug report; TeleportService has been failing a lot recently. This is a graph of all server-side teleport errors from the past 30 days in my experience:
However, my matchmaking system is global so players are teleported to the match server using the code above from different lobby servers (the game’s universe has a lobby place and match place structure).
Yes, there were more teleport failures in the past 20+ days, which we think they should have been fixed at the end of last Friday (Oct 25).
The error messages you posted are specifically about HTTP limits, which happened even before a teleport request was made to the backend. These HTTP limits can be from previous teleport requests, or requests to the other endpoints in the backend, like datastore requests, etc.
Do you have any other logs before those about “number of API requests/minute exceeded limit for HTTP API throttle”?
I only have access to logs since September 29th, but from what I can tell there’s only been the occasional similar error. However, no more than 1 per day at max.
Nothing much has changed on my end in terms of API calls, and as stated in the original post the game’s systems were able to sustain 7,000 CCU at one point with no issues like these.
The services I use are:
TeleportService
MemoryStoreService
MessagingService
DataStoreService
… well, pretty much everything I suppose.
Could there have been made any changes to the API throttling on your end recently?
It’s worth noting that only TeleportService calls are being throttled.
Do you have any idea when this started happening to your experience? By checking the history of the settings I don’t see anything related to API throttling recently.
I’m still wondering why your experience has so many teleport requests that could trigger the throttling. I haven’t seen this issue from any other experiences yet. Meanwhile, I’ll start looking into other options to mitigate the throttling for teleport service specifically, will post when we figure out a plan that can be shared here.
Avallachi found the root cause in the module that calls TeleportService:ReserveServer() which is a HTTP request and should be wrapped in a protected call. The ticket can be closed now.
