We’re using MessagingService via Open Cloud to send matchmaking information from an external system. We are well within limits for our game (based on rough estimates of our message count and using the budget based on our average server count), and all publishes through Open Cloud succeed. However, when attempting to use PublishAsync on the same topic in a server, we get budgeting errors like the one below:
This error never disappears (and will constantly say we are over the limit) and we are unable to use PublishAsync on this specific topic. I’d like to note that other topics that don’t use Open Cloud work completely fine:
…so it is acting as if we are exceeding the received per topic limit, which for our game and this specific topic we are well under.
As a side note, if I attempt to publish to the topic twice within 30 seconds, I get this behavior:
Hi, thanks for the report! I believe the second screenshot is showing the time before you can publish another message (19 seconds), the second publish request looks like it is 3 seconds after the first? and that is why we see the second error please retry later.
I am taking a look into issues, could you provide me with some more information around your usage pattern? How often are you publishing from open cloud and rcc to the same topic? (or are they going to different topics?). What about subscriptions? Thanks in advance!
Ahhh gotcha, I was under the impression that it was some sort of other limit we were hitting.
From OC we’re sending around anywhere between 80-250 messages per minute on average. We’re no longer sending messages via RCC so this topic is purely on OC now, behavior still continues. We don’t use other topics at present but I have tested that in production other topics work fine.
Every game server has a subscription, so at time of writing ~7000 subscriptions.
We have not been able to reproduce this bug on our end yet. However we did see a surge in the number of game servers for this universe between Apr 16 12:00 PM - 13:00 PM PST. During that period, the game server count spiked from ~6000 to 7200, and gradually fell back to ~6000. We did notice some requests got rate limited which may be caused by the game server count changes in a short period of time.
Could you help confirm if the error is still occurring today?
Just so I can confirm, given our experience’s size how many messages/min should we be expecting to have? It sounds odd that we hit any ratelimits at all; I did do the math myself but I’m wondering if I got it wrong.
I cannot seem to reproduce the error today in a public server as I did previously, but I can reproduce it consistently in a reserved server of a subplace:
We have been working on a feature to enhance our rate limiting logging, and just released it today. Could you help us reproduce the error again by sending several requests (spacing them at least 30 seconds apart so their log don’t get mixed up)? Please send these requests from the same universe you previously shared. We will then review the logs on our end and hopefully can identify the issue.
(redacted logs are the result of the messages, and not relevant to this report)
Not sure if this is related to the fact that there are less messages being published today via OC, but you can see here that it did successfully send some. It seems like if they are precisely 30 seconds apart, they get sent successfully, but any earlier and it triggers a ratelimit (since it only happened again after I accidentally sent one 3 seconds too early).
Worthy of note, today’s behavior is different to what was previously reported as in the screenshots before even with ~30 second spacing it would consistently ratelimit with much higher retry times.
Specifically, during the time of these requests, this universe attempted to deliver 652,045 messages to 7,550 servers, which exceeded the the limit of 40 + 80 * 7550 = 604,040 messages per minute.
We observe that your universe frequently operates at this load or higher. You are not getting error because currently we are not enforcing this top-level rate limit on Open Cloud.
Is this number the amount of messages the system generates in total to send to all 7550 servers (i.e. 1 message becomes 7550)? And if this is the case, does sending messages through OC guarantee delivery even if it would go over this limit (since it doesn’t give any error when we publish)?
652,045 is the number of message deliveries in total. For example, if a single message is successfully delivered to 7550 servers, that counts as 7550 deliveries.
Currently we are not enforcing a topic-level limit on Open Cloud (per documented in Messaging usage guide | Documentation - Roblox Creator Hub); only a universe-level limit is enforced at this time. That should explain why you are not getting errors when using Open Cloud.
Okay, that makes a bit more sense. My understanding of that limit was the number of messages you could have sent to that topic from RCC, rather than the number of deliveries. (I was going to say, at worst we sent ~1k during a peak time!)
It seems a little punishing to me for the limits to be based off of deliveries, but I understand why it is that way and it’s feedback for a different topic regardless.
Thank you for the clarification and assistance, it’s been much appreciated!
