Hey Roblox team,
We’re having the same problem. We have a big event coming up this week and this issue is really worrying me because this happens repeatedly whenever many users join our experience. I hope this is resolved quickly before then.
Universe ID: 5962001419
Teleport From place ID: 17421919489
Teleport To place ID: 17422028416, 17422050968, 17422063658 (one of them)
When it happened: 6/22/2024 around 7:00 AM (PDT)
This is also happening in my game over at Little Warzone [INDEV] - Ants, Bugs and More! - Roblox.
TeleportService seems to randomly break in given lobby servers, spewing out the HTTP 429 error without the game actually hitting any ratelimits. When it does break in a given server, it stays like that permanently in said server, as verified by my output.
Since people can’t get from the lobby server to the actual places, this issue is pretty critical for my game. I’ve mitigated it a little bit by making lobby servers smaller (3 players per server), so it affects less people if encountered, but it still has a major effect. A player might assume the game’s broken if they can’t get to the other place and leave my game as a result. It’s directly affecting my metrics.
Over the past 30 days, I’ve had 2.2k errors from this issue. 2.2k lost players, assuming they left after this. Not good.
Thanks for the reports everyone. We looked into it and found a request throttle was accidentally being shared across multiple game instances. That’s fixed now. Please let us know if you still experience this issue
Thanks! Any news on 429s not firing TeleportInitFailed?
I’ll be keeping an eye on the graph to see if I see any further errors, but @bvetterdays above is correct in mentioning that the 429s are not firing TeleportInitFailed which is an issue of its own.
Any error that leads to a teleport being cancelled or left in an incomplete state should raise this event, and that’s an issue of its own that must be resolved regardless of the fix that has been applied here.
I appreciate the throttling fix being applied, but I think y’all ultimately need to do a better job at collecting metrics on TeleportService related outages on your end because it has happened on numerous occasions that I have had to create bug report posts for problems that should be visible via your own metrics.
It’s frustrating to see mid-week changes made like this, without any active metrics on your end to pick up on the issue, and then not resolving it til Monday which has a direct monetisation & algorithm impact over the weekend.
Policy should change around this rather than a simple tweak of some nginx config & moving on.
Continuing to see these errors, albeit not at the same rate as before. Once again taking the opportunity to mention these errors cancel the teleports without raising an error, so we can’t handle them or re-try them in-game.
@bvetterdays @Brian1KB I’m talking to the team that owns TeleportInitFailed to see if we can improve it
@Brian1KB Can you post screenshots / logs? Do you have a sample Job ID and rough timestamp? The original issue I fixed has not reoccurred in the last two days so this is something else
Apologies for the slow reply, I had some issues with my internal metrics. I’m not seeing this at the same frequency I was prior, but this does continue to be a recurring issue I’m seeing in my logs. Here are job IDs & timestamps for that error in the logs.
“Time”,“errorlog.distinct { jobid: , msg: }”
2024-07-15 18:51:44,b7ad1085-7a98-4efd-bc20-c7a4f8270edc
2024-07-15 18:52:07,9831ab86-eb1a-4d7d-835a-add4d3c33527
2024-07-15 19:28:39,d5e7e5f9-9daa-40c9-97cf-608fa8454289
2024-07-15 19:40:35,a097acdb-7012-4dd4-95ea-851d22e4abe3
2024-07-15 19:50:02,bb3260f8-848c-44ce-8772-46f4e1231fe9
2024-07-15 20:50:51,1d851dfa-9203-4484-b200-b709c02fc29e
2024-07-15 20:58:21,b571cd7c-26b6-4dcc-9dfe-bb3bf46440b8
2024-07-15 21:28:00,4dc91922-26d8-481b-9714-03f94264da56
2024-07-15 21:35:43,2dd03584-ea04-45d2-9219-f36de5588d1b
2024-07-15 22:00:28,c6a76e3c-9d30-43e0-b1c8-90343db7a4a1
2024-07-15 22:30:29,cc9e0d75-314b-4c4c-9d5d-5ba87c744617
2024-07-15 22:45:10,36308f0b-4d88-4d8d-b5e8-86c862f13be6
2024-07-15 23:00:03,cd511a95-5d26-48b5-8665-233105280457
2024-07-15 23:00:04,2a0d5fcd-d7c2-422d-8c84-44401429d6c6
2024-07-15 23:00:06,fa15244a-8920-4782-9168-f3202c50f997
2024-07-15 23:00:07,b14d429b-8008-4cef-af3f-f71becf802ab
2024-07-15 23:00:09,ace9752e-57e9-47bd-98c0-baf868ceecd5
2024-07-15 23:54:48,9bf4e53c-3c34-416a-9274-05aaccb58a26
2024-07-16 01:04:08,3c62d533-127b-4e54-98b1-5d9aea827c5f
2024-07-16 01:07:48,670d6b20-d6ad-434a-9947-f5615bf894d1
2024-07-16 01:24:14,5a3a1247-452b-4da5-a083-2f5cc5fc31b1
2024-07-16 02:10:58,4eed93b6-71f8-41de-b46d-8bdb684f0b63
2024-07-16 02:17:25,0a182211-65ed-49e7-bca8-340c69c8392e
2024-07-16 03:08:39,76567f38-a5d5-41a2-a641-46098fb04c88
2024-07-16 03:31:20,90966f0e-c5f9-4208-9584-5bd2f7972ce8
2024-07-16 03:45:11,8339a596-5900-4381-9bae-e4e7778925fe
2024-07-16 04:34:11,d1b6184e-282f-417b-9fda-7a15286e4ccf
2024-07-16 05:12:31,2f435206-bb4d-40a1-8e0f-3bf9f261855f
2024-07-16 05:29:49,c8f57edd-9c4d-4ff5-b6af-11bb190f0c34
2024-07-16 06:15:49,a6c3d4e0-3846-4dba-9ac2-b91b56528795
2024-07-16 07:24:44,8f6860f9-0a29-4151-abd6-5edcab6ac841
2024-07-16 07:46:23,6b8345f2-37db-45e3-82d2-267ae7eb75e5
2024-07-16 08:02:23,182fd07a-be1a-46d0-958b-c208d9fab8bc
2024-07-16 08:06:43,33fc929d-a735-478a-bba5-40e159fbb812
2024-07-16 08:20:30,8f6860f9-0a29-4151-abd6-5edcab6ac841
2024-07-16 08:32:19,a1640c5c-082c-4392-9ab6-260f19b01984
2024-07-16 09:40:49,7659c734-fcfd-451a-b147-3fc0aafda42a
2024-07-16 10:04:48,8367eb84-76a0-4079-b841-f1e57dca0c59
2024-07-16 10:39:28,580d2ae5-c0a2-4e9b-8d1a-3e1a504d3d30
2024-07-16 11:33:37,5bab8d03-9f50-48b1-ad3b-7d5aa22d28d3
2024-07-16 13:02:55,11f1f3b8-674e-42e3-97eb-baee984aebda
2024-07-16 13:22:59,120785b0-2ba2-4199-a789-0f7a027ed04d
2024-07-16 14:01:10,f2bd15e8-93e1-4b2d-a710-e98131c03455
2024-07-16 15:20:54,55e3e761-0f84-47c3-8116-c6f88067d19c
2024-07-16 15:30:46,6e1a4080-a75e-4005-bb15-72d27dabf178
2024-07-16 15:53:29,3d015f66-2a79-44ac-9bda-25511186a1b5
2024-07-16 16:18:43,0695100b-e6fd-4f6a-aea4-57f4b1ea4df5
2024-07-16 17:14:08,b25e1909-f08f-4477-9904-c065c5a00d40
2024-07-16 17:40:11,200f8fe0-90b0-4061-9307-bda5d3d45aa8
2024-07-16 17:59:11,bec148c1-e450-4660-b1cf-3aa1f823669c
Thanks for the timestamps and Job IDs. Took a look at one and it was caused by a transient network failure. The original throttling issue hasn’t reoccurred since the fix was applied a few weeks ago. For transient network issues, properly raising TeleportInitFailed as suggested earlier will be helpful. I was able to get a ticket filed for it. No ETA, but at least it’s being tracked now. cc @bvetterdays
In my game plane crazy, i’ve noticed teleporting can also take an extremely long time.
I’ve even experienced it myself 2 times already so far where it wouldn’t teleport me for 60 seconds straight erroring with (teleport already in process) until finally teleporting me.
This is really bad when updating my game as all players are teleported to the updating place.
If it takes too long for a server to shutdown, it will cause me to lose a lot of players during an update.
Just to clarify, are you teleporting players on shutdown to another Place, waiting for some time, and then teleporting them back? If so, that update pattern shouldn’t be used anymore. When you shut down your servers for a game update, Roblox will spin up replacement servers and only shut down the old ones once the new ones are ready. Players will be automatically teleported to those new servers after your BindToClose callbacks have completed. If you need to customize the teleport logic (e.g. adding some TeleportData), then doing a Teleport in BindToClose to the same PlaceId is sufficient.
For more information, see:
I never knew about this, i only knew about the button “migrate to latest version” under the game but that isnt easy to combine as it shuts down the old servers without teleporting the players i think.
Is it possible to automate this via a script and how long does it take for the players to get teleported and the new servers to open?
I’ve also noticed in the documentation, players seem to get prompted with an option to leave or rejoin, is it possible to remove this option? It could mess with my game’s code if they stay ingame for too long and could make me lose out on potentional users playing the game and clicking the wrong button by mistake.
In my game i have to warn the players 2 minutes ahead of time before a shutdown to prevent savedata loss before teleporting them.
it shuts down the old servers without teleporting the players i think
As of a few months ago that should no longer happen (see second link above). If it does, please file a bug report
Is it possible to automate this via a script and how long does it take for the players to get teleported and the new servers to open?
You can teleport in BindToClose to the same PlaceId, but note that this is already the default behavior (again, see second link above). Replacement servers will already be ready, so the join should be as fast as clicking the Play Button (see first link above → Prewarming Updated Servers)
I’ve also noticed in the documentation, players seem to get prompted with an option to leave or rejoin, is it possible to remove this option?
Users should no longer see this (second link again 
). Do you have a link to the docs showing this? We’ll need to get it updated
In my game i have to warn the players 2 minutes ahead of time before a shutdown to prevent savedata loss before teleporting them.
Without knowing the specifics of your game, saving their data in BindToClose callbacks is the way to prevent data loss. I know this doesn’t work for everyone though (as in, your game’s logic might not work well with this)
Let me know if you have more questions or run into any issues!
As of a few months ago that should no longer happen (see second link above). If it does, please file a bug report
Is this the same for migrate to latest version button, or will i have to go to the creator dashboard instead?
Users should no longer see this (second link again
Without knowing the specifics of your game, saving their data in BindToClose callbacks is the way to prevent data loss. I know this doesn’t work for everyone though (as in, your game’s logic might not work well with this)
My game allows players to disable the autosave slot because of data limits, they have to prepare for the server shutdown to prevent dataloss when they’re in the middle of re-saving something.
Is it possible to trigger the restart servers effects via a script ingame instead?
I dont want to be waiting for 2 minutes to perfectly time the migrate button each time i update my game…
Is this the same for migrate to latest version button, or will i have to go to the creator dashboard instead?
All of the various “shut down all my servers” buttons should have auto-reconnect at this point (please file a bug report if you see otherwise!). We’ve consolidated all of them to use the same process behind the scenes. The only difference now is whether we look at the Place Version when deciding whether to shut down a game (Shutdown All Servers will shut down everything, Restart Servers for Update / Migrate to Latest Update will only shut down servers that are not running on the latest published Place Version)
Do you have a link to the docs showing this? We’ll need to get it updated
Excellent, thanks for underlining! Will get that line updated to say that users will automatically reconnect.
Is it possible to trigger the restart servers effects via a script ingame instead?
Not possible today, unfortunately. We’ve received this request before, not sure if there’s a Feature Request thread to track it
I have switched over to using the restart servers button for a while but it turns out to be unusable for me.
I have just accidently shut down all servers instead of restarting them for updates because i have to constantly press a button manually for this while i used to just change a simple number in my system upwards.
The buttons are really close to eachother and look very similar, it got me very confused and caused a loss in playercount as a result.
Before:
After 15 minutes:
Looks like that specific Shutdown button is using the old flow without auto-reconnect. I’ll file an internal bug report for that
There needs to be a way to automate this
You can use the following Open Cloud API: Universe | Documentation - Roblox Creator Hub
Hello, we updated the “Shutdown All Servers” button to auto reconnect players after the server has been shutdown. Thanks for your patience with this issue.
Bump, my game “Exodus” which is currently in private testing, currently has extremely long teleportation times (5-20+ minutes) from the main menu place to the main game place. I have no idea as to what could’ve triggered it as I haven’t touched the script in over 2 months and it started acting up only around 2 days ago.
It also seems like when I shutdown all servers while in the queue for teleportation, it teleports me immediately.
