Make messaging service more scalable

As a Roblox developer, it is currently too hard to make effective cross server consistent communication in any game with a player base, this is due to the current messaging service restrictions being too restrictive and too badly scaled for them to be useful for anything short of trivial message systems used to tell players that an update is coming.

Currently our game Blox royale uses messaging service to handle a cross server matchmaking system, we though at great length wether or not to host our own servers or use playfab, but in the end we decided to give messaging service a chance, and a week later after some unnecessarily complex scripts it was ready and we put it into production.

In European times when we currently have ~40-60 players this system works perfectly fine and we have no problems with it, however when we start to get to the ~80-150 range we start to see a bunch of errors in relation to surpassing messaging service limits being logged, something which is currently severely limiting the scalability of our game.

I think from the fact we have no problems at non-peak times there is enough evidence to say there are just about enough calls to handle a system like this entirely on roblox, and despite a lot of optimisation of both the data being sent and the amount of calls we still unable to scale with our player base at peak times.

Im not necessarily asking for an increase to the base limits, but I feel for messaging service to be a viable method for cross server communication in the future for roblox it needs to be able to scale with a game’s player count, and therefor I would like to request that all the limits increase more dramatically with both amount of players in the server and total amount of active servers.

If Roblox is able to address this issue, it would improve my development experience because messaging service will finally be a viable tool in any game, not just test demos and may allow for some really innovative and creative experiences on roblox.

14 Likes

If I recall correctly, fast write times to datastores is on the roadmap, so matchmaking would not use Messaging Service but datastores.

2 Likes

(In hindsight I think this post got a bit big, let’s hope the engineers, or anyone else, don’t hate me after having to read a college essay, sorry for any school-related traumas I might have brought back with this post)

Unfortunately the roadmap has yet to be updated to its 2021 version (if I had to guess that probably either means Roblox is still considering a few things to be placed on the roadmap, or they’re going to update it late/not at all this year)

In the previous few years of using the roadmap Roblox has made changes to it between years for the couple features that might’ve run into significant technical or performance issues and can’t be realistically implemented.

I agree that MessagingService should have slightly looser restrictions, however, I am a bit worried about global performance impact. If you take into account the fact that there are thousands, probably more like a few hundred thousand active Roblox servers at once that can all be relaying messages to one another, the idea of loosening limits too much sounds like it could be dangerous.

I’m not sure the architecture of how MessagingService is set up under the hood, but, from my understanding every server in your game connects to a relay server (if it can be contacted an error is thrown which is why I know there is some sort of existence of a relay server). That relay server could be unique to your game’s universe I suspect, which might have been the inspiration for UniverseScripts (or maybe the other way around. I still think those would be cool as hell, I’m sad to see they unfortunately never made it to fruition for some reason or another), or perhaps there could be one or more centralized relay servers that multiple games might communicate with.

I suspect that, depending on the architecture underneath, with games as large as Adopt Me that have a history of being the number one contributor to unleashing total site-wide destruction onto Roblox, the impact of scaling up MessagingService could be a lot.

Thankfully though, I do have some experience in the area (particularly real time networking) to suggest something. While its likely extremely difficult to send a lot of small messages at once, its actually much, much easier to send big messages very very slowly. There are a few features that work like this on Roblox already, iirc DataStores is one of them but I could be wrong, some features can queue up a lot of requests and send them in one go.

By allowing a “fallback” limit for when your game reaches the existing limits, Roblox could simply allow for messages to queue and send them in large batches to the relay server(s), ideally having them send out to the target servers in roughly the same relative times they were received in (which would fill the time between potentially receiving the next batch). That would introduce a ton of latency when sending messages past the normal limits, which could, for example, range from a few seconds, to maybe even closer to a minute or more, however, that would still allow for real time communication and would actually allow for significantly more data to be sent at the benefit of potentially even improving performance depending on how this new limit might be tuned. (Plus of course, timings can be tuned automatically so that they can scale up with larger games too)

This might actually even be able to be extended to the existing limits within the game, where a game might batch 1-2 seconds worth of messages to be sent at once by default. Assuming the current limits were in that case adjusted to better fit, if you send a lot of tiny messages this way, it would really only cost about the same as one message. You can only send 10kb in one message iirc, so, say you send a hundred 0.1kb messages. That totals to 10kb of data still, and, would only incur the additional cost of keeping track of the relative times between the messages on top of the time the batch was sent which really only adds a few bytes at most, which comes out to have negligible effects on size, maybe at most adding an additional 1kb of data.

3 Likes