I increment and decrement a counter before and after pcall-ing PublishAsync. I have a server where the counter is stuck.
Upon restarting the server, I get a few requests in, then it gets stuck yielding forever again. This completely breaks my experience.
Expected behavior
MessagingService:PublishAsync calls should eventually time out or error. Infinite yields like this are unacceptable, even during an outage.
It seems every few weeks there’s a service issue like this and it cripples my player count. It’s okay for services to go down once in a while, but game servers must be able to recover! Players just get stuck in the lobby indefinitely until I manually restart them.
Is there any way to reproduce this issue or does it just happen randomly? Would love to know if there’s a simple repro so I can patch this in my games to avoid it breaking essential systems.
It seems to happen randomly, but is always the result of an outage (or happens within a few days of one.) To simulate the infinite yield you could imagine replacing MessagingService:PublicAsync calls with Instance.new("BindableEvent").Event:Wait(). I’m setting up my system to detect when these calls yield for more than a minute, and report these detections and whether they eventually recover. Once I understand it better I can make the lobby servers automatically restart when these infinite yields are detected, although maybe it would be better to just retry the request with a custom timeout. APIs infinitely yielding like this is just broken though. It’s almost certainly a memory leak, and debugging it is a waste of time.
I started tracking when PublishAsync yields for a long time. It seems to happen frequently.
When a request takes 10 seconds, it warns “Messaging Service long yield detected”, and I display a “Services Down: MessagingService” message for players.
When a request takes 60 seconds, it warns “Messaging Service infinite yield detected”, and I prepare the server for a restart.
Looks like there are 81 restarts within the past day, and it took longer than 10 seconds 198 times. These requests are needed for players to join servers and these long waits can break the experience. I should clarify that private messaging also uses this message manager module, and the message requests for target servers are pooled dynamically, so these numbers don’t necessarily correspond 1:1 with players getting stuck on a loading screen, but I know it does happen.
I should also note that I made it warn “MessagingService long yield recovered” when a yield that takes more than 10 seconds recovers and stops yielding, but based on my stats, none of them ever recover if they take more than 10 seconds.
Hey everyone. Waned to give a quick update here. We are looking into potential disconnect issues that can be exacerbated by sitewide issues. We have a fix in mind but this may take a little bit to implement and launch. I will share here when this is going live.
@Tomarty can you check if you had this issue happen in the past 3 days? We think we found what could have caused the issue, which has been resolved on Friday. We would appreciate it if you could check if it happened to your experience again. If so can you provide me your experience ID so I can investigate further? Thanks.
It appears to have stopped after 2023/10/27 at 12:00PM (assuming the Error Report tab filter works, as it’s a bit confusing in relation to the date range.)
Thanks for confirming this has stopped happening. We’ve set up alerts for the issue and already started working on a fix to prevent this issue from happening in the future.
