For now I basically just double wrapped the calls in a pcall and a coroutine. That way (I think) it doesn’t yield on the server and cause everything else to stop working.
More generally would just recommend not having essential systems rely on messagingservice to function. Even global announcements can probably be done with memory stores with a bit of effort.
9 Likes
This has affected my game, Pastriez Bakery. Our training center uses messagingservice to create and manage its training servers.
10 Likes
Hey everyone. MessagingService is recovered now
18 Likes
Hi, I think we’d like to know why this has been happening and how it has been resolved, I think we deserve transparency in regards to the situation.
8 Likes
This was an absolutely breaking change for Arsenal and it resulted in ALL new servers being unable to start for what seems like a 7 hour period. It has resulted in our average session time dropping by HALF for the outage and will have resulted in who knows how much lost revenue and ruined first time user impressions (ESPECIALLY important since the recent release of PlayStation).
This was not a minor bug, and I’m hoping we get some insight as to how and why this happened, alongside what measures have been taken to ensure they don’t happen again. It’s random breaking changes like this that plague our games and have always been a source of worry regarding the platform.
9 Likes
Very well said. Engineers seem to underestimate the importance of uptime for services like MessagingService. For games that depend on it and are forced to turn away new players during downtimes, it can severely hurt their earnings and potentially affect their long-term success. Considering the recurrent issues in recent months, I’ve even been contemplating using HttpService with long-polling as an alternative to MessagingService (which is a last resort, mind you).
I hope Roblox can offer some clearer insight into the reason behind these disruptions and the measures they’re taking to prevent them from happening again.
5 Likes
Im still experiencing issues with messaging service. Now instead of throwing an error, it is just infinitely yielding when attempting to subscribe/publish a connection. Please fix this ASAP as this can be game breaking.
3 Likes
I can’t reproduce this. Are you testing it in a live game server?
5 Likes
Seems to have been fixed now. Still was very concerning considering the fact that it yielded infinitely with no errors or warnings.
5 Likes
Hey everyone. I wanted to close the loop here and give an explanation on what happened.
At 7:30PT MessagingService saw a massive increase in publishes. This was in multiple orders of magnitude. The system that handles influx struggled under the load then began failing to limit globally causing the outage in both publishes and subscribes.
We know this service is critical to many of your games and there is already planned work to catch and keep these types of issues from causing widespread outages. We have recently been putting a lot more effort into improving MessagingService and I hope to share some awesome news with you all soon.
We have an awesome group of creators using MessagingService in this thread. Please feel free to comment on the things you run into directly with the system in this thread or message me directly. I am also interested in use cases that MessagingService could solve for you in the future if certain changes are made.
11 Likes
Thanks a lot! This insight is greatly appreciated and it’s good to hear that this is actively being worked on. Excited to see what you guys have planned for MessagingService in the future. My only suggestion right now would be to increase the publish limits as well as the max data size (currently 1kB). 
6 Likes
Sorry for the second reply - there’s two more things I wanted to mention here.
1. Could we have some clarity on if MessagingService tries to resubscribe to topics it disconnects from (usually due to an outage)? I’ve seen errors along the lines of “Resubscribe failed” which leads me to think it does try to resubscribe, however it seems it gives up after some time. I’d suggest it tries to resubscribe periodically (maybe every 1-2 minutes or so), because with however it currently works, we’re forced to reboot certain servers after MessagingService outages because they’ve disconnected crucial subscriptions.
2. Not sure if this is your field, but there’s currently no documentation regarding MessagingService rate limits with Open Cloud APIs. I asked a while back here and an engineer said they’d get back to me but never did. Even if you could state them here (if you know them) that would be great.
5 Likes
Thanks for the update!
I would love to see a section in developer stats similar to MemoryStores but for MessagingService.
7 Likes
- If a subscription drops the MessagingService will try to re-subscribe once for each topic. There are improvements we can make here.
- 50 + 5 * players in universe per minute. Documentation is lacking here. We will update it.
6 Likes
Thanks a lot for the assistance! I have two more questions I had earlier but only recalled now (last ones, I promise!). 
Are rate limits per API key or per experience? I’ve observed that for Open Cloud DataStores, it’s per API key. I’m curious if this holds true for the MessagingService, however it scales based on CCU so I understand if not.
Would it be possible for the Open Cloud MessagingService to return errors if a message fails to deliver (due to downtime or other reasons)? My game features a global events system where players buy experience-wide events. These events then communicate with my external API, which subsequently requests back to MessagingService to broadcast the event across all servers post-validation. The issue is the API signifies a status code of 200 even when the message doesn’t get delivered. This hinders the automated refund process for players during service interruptions and causes in-game currency loss.
1 Like
Limits are per universe for MessagingService
I will look into your second point to see if we can make errors more visible
1 Like
Can we get a function that returns how much budget we have for different api calls? similar to datastore budget
2 Likes
I think :SubscribeAsync() is having problems. In the past few days, it would randomly hang indefinitely in some servers. I have a print statement right before subscribing (which gets printed) but nothing after the :SubscribeAsync() call shows up
2 Likes
This has been addressed: MessagingService:PublishAsync yields forever (2023-10-22) - #8 by jjwu_play
1 Like