Thanks! Any news on 429s not firing TeleportInitFailed?
1 Like
Iāll be keeping an eye on the graph to see if I see any further errors, but @bvetterdays above is correct in mentioning that the 429s are not firing TeleportInitFailed which is an issue of its own.
Any error that leads to a teleport being cancelled or left in an incomplete state should raise this event, and thatās an issue of its own that must be resolved regardless of the fix that has been applied here.
I appreciate the throttling fix being applied, but I think yāall ultimately need to do a better job at collecting metrics on TeleportService related outages on your end because it has happened on numerous occasions that I have had to create bug report posts for problems that should be visible via your own metrics.
Itās frustrating to see mid-week changes made like this, without any active metrics on your end to pick up on the issue, and then not resolving it til Monday which has a direct monetisation & algorithm impact over the weekend.
Policy should change around this rather than a simple tweak of some nginx config & moving on.
3 Likes
Continuing to see these errors, albeit not at the same rate as before. Once again taking the opportunity to mention these errors cancel the teleports without raising an error, so we canāt handle them or re-try them in-game.
2 Likes
@bvetterdays @Brian1KB Iām talking to the team that owns TeleportInitFailed to see if we can improve it
@Brian1KB Can you post screenshots / logs? Do you have a sample Job ID and rough timestamp? The original issue I fixed has not reoccurred in the last two days so this is something else
1 Like
Apologies for the slow reply, I had some issues with my internal metrics. Iām not seeing this at the same frequency I was prior, but this does continue to be a recurring issue Iām seeing in my logs. Here are job IDs & timestamps for that error in the logs.
2 Likes
āTimeā,āerrorlog.distinct { jobid: , msg: }ā
2024-07-15 18:51:44,b7ad1085-7a98-4efd-bc20-c7a4f8270edc
2024-07-15 18:52:07,9831ab86-eb1a-4d7d-835a-add4d3c33527
2024-07-15 19:28:39,d5e7e5f9-9daa-40c9-97cf-608fa8454289
2024-07-15 19:40:35,a097acdb-7012-4dd4-95ea-851d22e4abe3
2024-07-15 19:50:02,bb3260f8-848c-44ce-8772-46f4e1231fe9
2024-07-15 20:50:51,1d851dfa-9203-4484-b200-b709c02fc29e
2024-07-15 20:58:21,b571cd7c-26b6-4dcc-9dfe-bb3bf46440b8
2024-07-15 21:28:00,4dc91922-26d8-481b-9714-03f94264da56
2024-07-15 21:35:43,2dd03584-ea04-45d2-9219-f36de5588d1b
2024-07-15 22:00:28,c6a76e3c-9d30-43e0-b1c8-90343db7a4a1
2024-07-15 22:30:29,cc9e0d75-314b-4c4c-9d5d-5ba87c744617
2024-07-15 22:45:10,36308f0b-4d88-4d8d-b5e8-86c862f13be6
2024-07-15 23:00:03,cd511a95-5d26-48b5-8665-233105280457
2024-07-15 23:00:04,2a0d5fcd-d7c2-422d-8c84-44401429d6c6
2024-07-15 23:00:06,fa15244a-8920-4782-9168-f3202c50f997
2024-07-15 23:00:07,b14d429b-8008-4cef-af3f-f71becf802ab
2024-07-15 23:00:09,ace9752e-57e9-47bd-98c0-baf868ceecd5
2024-07-15 23:54:48,9bf4e53c-3c34-416a-9274-05aaccb58a26
2024-07-16 01:04:08,3c62d533-127b-4e54-98b1-5d9aea827c5f
2024-07-16 01:07:48,670d6b20-d6ad-434a-9947-f5615bf894d1
2024-07-16 01:24:14,5a3a1247-452b-4da5-a083-2f5cc5fc31b1
2024-07-16 02:10:58,4eed93b6-71f8-41de-b46d-8bdb684f0b63
2024-07-16 02:17:25,0a182211-65ed-49e7-bca8-340c69c8392e
2024-07-16 03:08:39,76567f38-a5d5-41a2-a641-46098fb04c88
2024-07-16 03:31:20,90966f0e-c5f9-4208-9584-5bd2f7972ce8
2024-07-16 03:45:11,8339a596-5900-4381-9bae-e4e7778925fe
2024-07-16 04:34:11,d1b6184e-282f-417b-9fda-7a15286e4ccf
2024-07-16 05:12:31,2f435206-bb4d-40a1-8e0f-3bf9f261855f
2024-07-16 05:29:49,c8f57edd-9c4d-4ff5-b6af-11bb190f0c34
2024-07-16 06:15:49,a6c3d4e0-3846-4dba-9ac2-b91b56528795
2024-07-16 07:24:44,8f6860f9-0a29-4151-abd6-5edcab6ac841
2024-07-16 07:46:23,6b8345f2-37db-45e3-82d2-267ae7eb75e5
2024-07-16 08:02:23,182fd07a-be1a-46d0-958b-c208d9fab8bc
2024-07-16 08:06:43,33fc929d-a735-478a-bba5-40e159fbb812
2024-07-16 08:20:30,8f6860f9-0a29-4151-abd6-5edcab6ac841
2024-07-16 08:32:19,a1640c5c-082c-4392-9ab6-260f19b01984
2024-07-16 09:40:49,7659c734-fcfd-451a-b147-3fc0aafda42a
2024-07-16 10:04:48,8367eb84-76a0-4079-b841-f1e57dca0c59
2024-07-16 10:39:28,580d2ae5-c0a2-4e9b-8d1a-3e1a504d3d30
2024-07-16 11:33:37,5bab8d03-9f50-48b1-ad3b-7d5aa22d28d3
2024-07-16 13:02:55,11f1f3b8-674e-42e3-97eb-baee984aebda
2024-07-16 13:22:59,120785b0-2ba2-4199-a789-0f7a027ed04d
2024-07-16 14:01:10,f2bd15e8-93e1-4b2d-a710-e98131c03455
2024-07-16 15:20:54,55e3e761-0f84-47c3-8116-c6f88067d19c
2024-07-16 15:30:46,6e1a4080-a75e-4005-bb15-72d27dabf178
2024-07-16 15:53:29,3d015f66-2a79-44ac-9bda-25511186a1b5
2024-07-16 16:18:43,0695100b-e6fd-4f6a-aea4-57f4b1ea4df5
2024-07-16 17:14:08,b25e1909-f08f-4477-9904-c065c5a00d40
2024-07-16 17:40:11,200f8fe0-90b0-4061-9307-bda5d3d45aa8
2024-07-16 17:59:11,bec148c1-e450-4660-b1cf-3aa1f823669c
3 Likes
Thanks for the timestamps and Job IDs. Took a look at one and it was caused by a transient network failure. The original throttling issue hasnāt reoccurred since the fix was applied a few weeks ago. For transient network issues, properly raising TeleportInitFailed as suggested earlier will be helpful. I was able to get a ticket filed for it. No ETA, but at least itās being tracked now. cc @bvetterdays
2 Likes
In my game plane crazy, iāve noticed teleporting can also take an extremely long time.
Iāve even experienced it myself 2 times already so far where it wouldnāt teleport me for 60 seconds straight erroring with (teleport already in process) until finally teleporting me.
This is really bad when updating my game as all players are teleported to the updating place.
If it takes too long for a server to shutdown, it will cause me to lose a lot of players during an update.
1 Like
Just to clarify, are you teleporting players on shutdown to another Place, waiting for some time, and then teleporting them back? If so, that update pattern shouldnāt be used anymore. When you shut down your servers for a game update, Roblox will spin up replacement servers and only shut down the old ones once the new ones are ready. Players will be automatically teleported to those new servers after your BindToClose callbacks have completed. If you need to customize the teleport logic (e.g. adding some TeleportData), then doing a Teleport in BindToClose to the same PlaceId is sufficient.
For more information, see:
I never knew about this, i only knew about the button āmigrate to latest versionā under the game but that isnt easy to combine as it shuts down the old servers without teleporting the players i think.
Is it possible to automate this via a script and how long does it take for the players to get teleported and the new servers to open?
Iāve also noticed in the documentation, players seem to get prompted with an option to leave or rejoin, is it possible to remove this option? It could mess with my gameās code if they stay ingame for too long and could make me lose out on potentional users playing the game and clicking the wrong button by mistake.
In my game i have to warn the players 2 minutes ahead of time before a shutdown to prevent savedata loss before teleporting them.
it shuts down the old servers without teleporting the players i think
As of a few months ago that should no longer happen (see second link above). If it does, please file a bug report
Is it possible to automate this via a script and how long does it take for the players to get teleported and the new servers to open?
You can teleport in BindToClose to the same PlaceId, but note that this is already the default behavior (again, see second link above). Replacement servers will already be ready, so the join should be as fast as clicking the Play Button (see first link above ā Prewarming Updated Servers)
Iāve also noticed in the documentation, players seem to get prompted with an option to leave or rejoin, is it possible to remove this option?
Users should no longer see this (second link again 
). Do you have a link to the docs showing this? Weāll need to get it updated
In my game i have to warn the players 2 minutes ahead of time before a shutdown to prevent savedata loss before teleporting them.
Without knowing the specifics of your game, saving their data in BindToClose callbacks is the way to prevent data loss. I know this doesnāt work for everyone though (as in, your gameās logic might not work well with this)
Let me know if you have more questions or run into any issues!
As of a few months ago that should no longer happen (see second link above). If it does, please file a bug report
Is this the same for migrate to latest version button, or will i have to go to the creator dashboard instead?
Users should no longer see this (second link again
Without knowing the specifics of your game, saving their data in BindToClose callbacks is the way to prevent data loss. I know this doesnāt work for everyone though (as in, your gameās logic might not work well with this)
My game allows players to disable the autosave slot because of data limits, they have to prepare for the server shutdown to prevent dataloss when theyāre in the middle of re-saving something.
Is it possible to trigger the restart servers effects via a script ingame instead?
I dont want to be waiting for 2 minutes to perfectly time the migrate button each time i update my gameā¦
2 Likes
Is this the same for migrate to latest version button, or will i have to go to the creator dashboard instead?
All of the various āshut down all my serversā buttons should have auto-reconnect at this point (please file a bug report if you see otherwise!). Weāve consolidated all of them to use the same process behind the scenes. The only difference now is whether we look at the Place Version when deciding whether to shut down a game (Shutdown All Servers will shut down everything, Restart Servers for Update / Migrate to Latest Update will only shut down servers that are not running on the latest published Place Version)
Do you have a link to the docs showing this? Weāll need to get it updated
Excellent, thanks for underlining! Will get that line updated to say that users will automatically reconnect.
Is it possible to trigger the restart servers effects via a script ingame instead?
Not possible today, unfortunately. Weāve received this request before, not sure if thereās a Feature Request thread to track it
2 Likes
I have switched over to using the restart servers button for a while but it turns out to be unusable for me.
I have just accidently shut down all servers instead of restarting them for updates because i have to constantly press a button manually for this while i used to just change a simple number in my system upwards.
The buttons are really close to eachother and look very similar, it got me very confused and caused a loss in playercount as a result.
There needs to be a way to automate this, until then i will keep using the old method i used to update servers as i never want this to happen again.
Before:
After 15 minutes:
2 Likes
Looks like that specific Shutdown button is using the old flow without auto-reconnect. Iāll file an internal bug report for that
There needs to be a way to automate this
You can use the following Open Cloud API: Universe | Documentation - Roblox Creator Hub
1 Like
Hello, we updated the āShutdown All Serversā button to auto reconnect players after the server has been shutdown. Thanks for your patience with this issue.
2 Likes
Bump, my game āExodusā which is currently in private testing, currently has extremely long teleportation times (5-20+ minutes) from the main menu place to the main game place. I have no idea as to what couldāve triggered it as I havenāt touched the script in over 2 months and it started acting up only around 2 days ago.
It also seems like when I shutdown all servers while in the queue for teleportation, it teleports me immediately.
6 Likes