We tried out MessagingService in an update yesterday - and it is working great - the only issue we are seeing the following some servers: “The game server has reached the allowed limit of active subscriptions.”
The wiki states that the max subscription limit is 5 + 2*numPlayers. We have two subscriptions per player, each of which is being disconnected as soon as the player leaves, so it should never be throwing the error unless the limits aren’t immediately adjusted with the player count - do you think that this could be happening? I will try wrapping :SubscribeAsync inside of a retry function and see if it throws the error after 3 attempts, but I’m mildly curious to know if anyone has experienced this issue as well.
Thank you so much for making a thread on this! For Vesteria, we do something similar, except we only create a single MessagingService subscription for each player, along with a few static subscriptions made by each server. According to the documentation, there’s no way we should be hitting the threshold but we end up doing just that.
We record subscription errors via GameAnalytics and have found that the threshold is inconsistent. There are spikes in errors saying that we’ve hit the limit and that just makes no sense given our implementation. We’ve gone over our code several times and hearing that you’ve had similar experiences makes me think that it’s not just us.
There is an ongoing issue where the subscription limit per game caps out at 10k – we’re in the process of removing this, but this should only impact popular games. What game are you using this in and how popular has it gotten?
Also, out of curiosity, why are you creating two subscriptions per player?
Hi @Seranok! This is for Royale High, one of the more popular games. We released an in-game texting system, so the first subscription is used on a player is used for when the player receives a message, and the second is used a “player is typing…” indicator. We could probably try using a single subscription for each player though in this particular case - but I don’t know if it would make a difference at this scale scale. Thanks so much for the update!
Based on that post I was assuming that the 10k subscription limit was intentional. I’m happy to hear that’s not the case! Once this is fixed the only limits will be subscriptions per game server and messages per server per minute and no more game-wide limits?
Hi @Seranok - we’ve started seeing this same error again over the last few days. We haven’t increased or changed the number of subscriptions in our game:
“The game server has reached the allowed limit of active subscriptions.”
@Ice7 The “TooManySubscriptions” error seems to be from the issue discussed earlier in the thread where somehow the number of actual subscriptions on the game server gets out of sync with how many we think there are from a throttling perspective. We’ve done some investigation on this but weren’t able to find anything obvious to explain what was going on. We still have some tickets open to continue the investigation, but they have gotten stuck behind some higher priority work. We do occasionally update the messaging service with bug fixes and small improvements and when we do it seems to reset the problem and the errors go away for a while.
The “Message Router Service disconnected” error happens when we do roll out an update to the messaging service and one was rolled out right at the time you saw the errors on Monday. These are normal and everything should automatically resubscribe and connect to another messaging service instance (server) when the one your game server was connected to goes down for the update.
Finally, to be clear, we still hope to get to the bottom of the erroneous “Too Many Subscriptions” errors. In the mean time, thanks for your patience!
The “Too Many Subscriptions” error seems to be causing memory to spike in those servers and essentially break entirely. I’ve been tracking servers that are using a high amount of memory, and in each of these servers I am seeing the “Too many subscriptions” errors. Everything seems to be processing super slowly in those servers in general, including Http Requests.
I am also seeing something really weird that may suggest subscriptions aren’t always getting disconnected. This might explain the “Too many subscriptions” error. I’m seeing the following error quite frequently:
As soon as the player leaves, it should be disconnecting the subscription, and the :SubscribeAsync event should never fire for that player until they join again. However, it looks like it is still processing, as it’s trying to index PlayerChannels[Player] on the subscription event, where PlayerChannels[Player] has already been set to nil, after the event was supposed to have been disconnected.
I don’t see how this could possibly occur unless the subscription takes some time to disconnect, or if it’s not getting disconnected entirely in some cases.
@Ice7 Sorry for the delayed response on this. For whatever reason I didn’t see your post until someone on our developer relations team brought it to my attention. We have been trying to track down a seeming bug where subscriptions don’t go away after being disconnected, but haven’t been able to reproduce the issue so far. Maybe with the information you have given we can finally do so. We will investigate and get back to you. Thanks for reporting this!
Hi @OutsideInSM! Just wondering if there’s been any luck on this issue - I’m seeing a correlation between high memory utilization in our start place where our traffic is highest and the MessagingService errors (likely due to the connections not getting disconnected). I think we will probably disable messagingservice there until it is solved.
Hmm, possibly it is due to the high memory usage. Perhaps it is if there is a memory overflow then some existing memory is overwritten. This is still an open investigation for us, and thanks for continuing to follow up with us.
Hey @Ice7, I’ve been seeing a similar issue for my game and I have a hunch as to what could be going on.
In the code sample you posted, you connect SubscribeAsync to PlayerAdded and set up your disconnect in PlayerRemoving. However, SubscribeAsync is a yielding method, and because of this it’s possible for PlayerRemoving to fire before SubscribeAsync ever returns if the player quickly leaves the game.
I believe this could be one reason you are seeing a “Too many subscriptions” problem: over time, although it is rare, enough players join the game and leave before SubscribeAsync returns that you get a buildup of subscribed topics that are never disconnected. Because the subscription limit is so strict, additional players will never again be able to have their topic subscriptions registered.
What I did to try to solve it on my end (although I haven’t pushed the update yet to see if it works) is to just add a check after SubscribeAsync returns to see if the player is no longer in the game, and if they aren’t, immediately disconnect the subscription.
Yeah, this would make sense as to why it’s happening in our Lobby mostly, the amount of time a player spends in there is relatively low typically, so if enough players’ subscriptions get set up after they leave, then they’ll remain connected indefinitely, causing the server’s memory to creep up over time, which is probably why I saw a correlation between high memory servers and the Too Many Subscriptions error. Thank you for the insight, I will try force disconnecting the subscription on the callback if the player is no longer in the game as well, and see what the results yield.
Just following up again on this. Your feedback has been greatly helpful to us to identify a potential root cause. We are still investigating and looking into potential fixes. Rest assured that we haven’t forgotten about this!
@Ice7 Thanks for your patience with this and help identifying this issue. One suggestion we have that may help is to add wait/retry logic to your SubscribeAsync calls. This is due to the fact that Disconnect makes an async call to unsubscribe/decrement the number of subscribed topics. Therefore the Disconnect call may have finished for the player, but the topic may not be fully unsubscribed immediately. This in turn may cause SubscribeAsync calls to fail with the TooManySubscribtions error.
We are still looking into the other issues that you have mentioned. Thanks again for your patience.
