Super simply put, a significant % of teleports are taking longer to go through than they should be. Players in my game are getting stuck on the loading screen when I attempt to engage a teleport for upwards of 60 seconds. I’ve included graphs which show the rate of arrival of players in to the reserved servers have dropped, and that the number of “teleport timeouts”, teleports which take >30 seconds has drastically increased over the past 18 hours. As a result, my session time has dropped which is also shown & the number of “game timeouts” aka games that could not start as they did not receive the minimum threshold of players has drastically risen.
3 Likes
Hey Brian1KB,
Thanks for reporting this, Could you please share the place id of the teleport from and the teleport to? I can take a look into this issue.
1 Like
Here’s the experience ID — on mobile right now so that’s all I’ve got, every place other than the primary place has teleports to it, and are experiencing the same issue. So from the primary place ID, to any of the “backend” places, many thanks.
1 Like
Got it! We are looking into this issue.
2 Likes
Hey Brian1KB, we looked into data at the day level and didn’t see an obvious trend of this issue, Are you able to post an updated metrics. We want to see whether it last the whole day or more like an error spike. Thanks a lot!
1 Like
Hey, sorry for the delay. I’m not 100% convinced that things are running as they should be:
This graph here tracks data on sessions ending, effectively, the blue line is showing that of players that had a teleport started, and got to the stage of being disconnected from the initial server instance, there has been a rise in the number that did not arrive on the target server.
Additionally, the yellow line tracks logins where a user joins our game and did not arrive via teleport to the location they were supposed to which we marked on their profile.
If that’s not entirely clear, the takeaway should be: of teleports being made with the TeleportService, there has been a significant increase in players that are not arriving at their target destination since 9pm (Irish Time) on the 19th.
Additionally, here is my graph of teleports where we have called the teleport service, and the players have not been kicked out of the initial instance after 15 seconds. There continues to be a noteworthy increase since the same time mentioned earlier.
It appears to me that a lot of this may be down to teleports not “engaging” as I would call it, this is the time period between the teleport method being called, and the actual player being disconnected & moved to the target server. The players appear to be getting stuck at that initial phase on the initial server.
I will say that whatever is going on here is fairly subtle in comparison to previous teleport outages, but there is a tangible impact from it too. Hopefully there’s enough data here for y’all to begin to track down things on your end.
Thanks for your time.
3 Likes
Thank you Brian1KB. The information you provided is very valuable! I will keep investigating. Please let me know if you find anything from your graph!
1 Like
Hey Brian1KB, sorry I have one more question, what is the timezone of your graph? UTC or PDT?
1 Like
I’m based in Ireland, so IST / GMT+1 at the moment with daylight savings
1 Like
In a similar time period i’ve had a huge amount of my teleportservice requests getting rate limited, I am not home to diagnose the issue but it is seemingly clearly something on Roblox’s end as I have not updated the relevant game in the past couple of weeks and the number of errors has skyrocketed from near 0 to about 90% of attempted teleports.
Teleport to ReservedServer failed : HTTP 429 (Too Many Requests)
(At 2PM on this graph I shut down the only running lobby server)
As far as I can see the game is attempting no more teleports than usual and hence my presumption is this is not an issue on my end, however as far as I know teleport service rate limits are not documented so it is possible something has changed there.
In my case, the game is small, and it seems the issue is specific to individual servers (I’ve already checked my retry after error logic and it does not cause any significant increase in teleport attempts)
As my game tends to only have one lobby server at a time, a server getting stuck unable to send any teleports is critical to me as it prevents anyone playing the game.
It is theoretically possible this is being caused by a malicious actor spamming all of my teleport related remote events however they are all rate limited at a rate where I don’t see how one user could be making any significantly out of the usual number of requests, and my logs of players in the lobby do not support this theory.
The experience ID in question is 3929033413
I’d appreciate any insight or ideas anyone can provide.
Thanks
2 Likes
Henry appears to be spot on in his post above – these errors are constantly streaming through the console on my initial place locations. Would really appreciate if we could get some insight in to the timeline on this issue.
1 Like
& some additional insight on this:
- TeleportAsync itself is not throwing an error when wrapped inside of a pcall when this occurs
- TeleportService.TeleportInitFailed is not fired when this error occurs, so we can’t actually attempt to re-try the teleport, or cancel it
It sure appears to me that this is the source of the problem.
1 Like
& one final addition of some metrics from the Error Report page, showing these errors from over the past few days.
A teleport really should never fail without firing the TeleportInitFailed, and it looks like it can happen in any of these circumstances.
1 Like
Joining in here to say this has been happening for me too. We have lobbies that are 1-player-servers where each player can be teleported into the actual game by clicking a join button, and we frequently have 429 errors spat out in the console when we literally have a cool-down on the join button to prevent this, which yields until an error is thrown through TeleportInitFailed or just infinitely assuming the player has been successfully teleported and no error has been thrown. This implies 429s are being thrown for no valid reason.
But as stated here, you’re again exactly right. We have something set up that logs a custom warning with the Enum.TeleportResult (or whatever this enum is called) that TeleportInitFailed passes each time it fires on the server, and we notice with 429s we just get the red error from Roblox itself, and no custom warning we’ve set up, meaning I can confirm that TeleportInitFailed does NOT fire if the error is a 429. This means that players who press the join button to get into a server will be stuck on an infinite cool-down because TeleportInitFailed isn’t being fired.
This is such an awesome thread; you guys have pinpointed every single issue we’ve been experiencing with TeleportService in our own games, thank you so much!!
2 Likes
Our game is having the same problem(http 429). Players are not able to teleport to game place… (it is temporarily fixed when restarting lobby place)
2 Likes
The Teleport to ReservedServer failed : HTTP 429 (Too Many Requests) error also occurred in my experience from 2 PM - 4 PM EST yesterday. Even after restarting the affected places these errors still occurred for a little while. I thought it was my code at first, but it seems like I’m not the only one experiencing this problem.
2 Likes
For anyone dealing with this over the weekend, the solution that I ended up going with until Roblox get in to the office on Monday is to use the LogService#MessageOut & filter for this specific error, and when a threshold is reached, I attempt to send all of the players in the server instance to another lobby to force it to shut down. This has been working for me. Not a great solution, but the best we can do.
2 Likes
Hey Roblox team,
We’re having the same problem. We have a big event coming up this week and this issue is really worrying me because this happens repeatedly whenever many users join our experience. I hope this is resolved quickly before then.
Universe ID: 5962001419
Teleport From place ID: 17421919489
Teleport To place ID: 17422028416, 17422050968, 17422063658 (one of them)
When it happened: 6/22/2024 around 7:00 AM (PDT)
1 Like
This is also happening in my game over at Little Warzone [INDEV] - Ants, Bugs and More! - Roblox.
TeleportService seems to randomly break in given lobby servers, spewing out the HTTP 429 error without the game actually hitting any ratelimits. When it does break in a given server, it stays like that permanently in said server, as verified by my output.
Since people can’t get from the lobby server to the actual places, this issue is pretty critical for my game. I’ve mitigated it a little bit by making lobby servers smaller (3 players per server), so it affects less people if encountered, but it still has a major effect. A player might assume the game’s broken if they can’t get to the other place and leave my game as a result. It’s directly affecting my metrics.
Over the past 30 days, I’ve had 2.2k errors from this issue. 2.2k lost players, assuming they left after this. Not good.
3 Likes
Thanks for the reports everyone. We looked into it and found a request throttle was accidentally being shared across multiple game instances. That’s fixed now. Please let us know if you still experience this issue
5 Likes