Preface: Empowering innovation.
Roblox has gone truly above and beyond what it used to be 6 years ago, and definitely for the best. The team at Roblox provided us with incredible tools to enhance the visual effects of our games at arguably low performance cost.
The introduction of PBR textures, material variants, better atmosphere, Future is Bright, Dynamic meshes, Skinned meshes, Rigging, Wind, and so on and so forth, has made possible what we could have only dreamed of in the span of a relatively short time, especially considering the maintenance cost and investment that those feature required and requires still. Truly, Roblox is in my opinion headed in a great direction, both for the players, the creators, and the company itself, but there is one area that has been lacking and that I feel has been overlooked now for a long time: servers.
Right now, our studio is hitting a brick wall which so far shows no cracks. The server’s memory limit of 6.25 Gigabytes used to be great years ago but with the new enhancement, it has now become insufficient.
I am usually not one to go forth and complain when something doesn’t grab me by the hand; in fact I think most programmers do like a certain amount of challenges from time to time. But with the new amazing tools that Roblox has provided us, it has been increasingly incredibly difficult now to use those features while also respecting the server’s limited memory. In fact, there are a lot of time where I’m completely puzzled as to what is there left for me to do better, in order to save again on more server memory.
Context:
Frontline: Karelia has been under development for 2 years now, with the initial release dating back from February 2022.
Initially, the game had only 3 maps, stored in ServerStorage, some of which were terrain regions with parts, some others were exclusively parts.
As time went on, we started adding more and more to our maps, in an effort to break records, in an effort to go truly beyond anything made on Roblox. We started putting an emphasis on map quality first and foremost, with great texture quality, using the full PBR capabilities, material variants, lightings, meshes, Intricate level design, fiddling with collision fidelities, hitboxes.
As of today, the studio is working on a truly huge update, update 1.5 brings in two whole new maps, new classes, additional material variants, more meshes for more details, etc.
But during our testing session, the server crashed after the maps were loaded 3 times, precisely 3. After fixing memory leaks, the issue was persisting.
So what’s the issue ?
Server caching, Physics Octree, Lack of memory management tools, and the 6.25 GBs memory limit.
- The first glaring issue right now which already digs a huge chunk of usable memory is the Physics octree. The Octree by itself is a necessity, but maps that are located in ServerStorage, where no physic calculation is done whatsoever, are fully loaded into the physics octree, or at least are loaded into the “PhysicsPart” category of the server memory. This should not be the case at all, all maps combined takes up to 800 MBs of memory which is never going to be used for any physics calculation whatsoever.
Here’s a screenshot of the server memory of an empty baseplate:
Here’s a screenshot of the server memory of the same empty baseplate with all maps added into the server storage:
This is studio, so we can easily halve that number for an accurate representation of what the actual server has in storage.
800MBs of data that is, apparently, never going to be used.
On top of that, I tried to no avail, to set the collision, touch, query properties of all the parts in said maps to false, in an effort to at least remove them for the PhysicsPart tree, with no success.
Maybe am I missing something ?
When the maps are cloned to be loaded into the workspace, that memory category can raise over 1.5 gigabytes, and thankfully when the map unloads, decreases back to 800MBs. But there’s a catch, or rather, there’s a cache:
- Workspace loaded maps (Cloned), which are then unloaded (Destroyed), are immediately cached by the server in the “UntrackedMemory” category. This is bad, really bad. All maps have their own quirks, with some of them having unique meshes that may not be reused in other maps, all maps have unions which are definitely not being reused elsewhere, and surprisingly despite the maps already existing in the server storage, the map gets fully cached when unloaded.
In the following screenshot you can see:
The initial untracked memory of the server during the map voting phase, which stagnates at 350 MBs:
Once our worst case scenario map loads in, the untracked memory raises up to 480 MBs, so far so good:
When the map unloads, and if we load the same map again, you can see a spike as the server caches the map as it unloads (unloading is deferred on this particular map, because of the huge data size it takes, despite our effort to minimize that impact). Once the server loads the same map again (using :Clone()), the server will reuse that cached data and the memory drops back to 480 MBs. Note the spike to 1.5 GBs.
But we already hit the wall; like we previously saw, the caching done by the server is detrimental to our project, because once another map is voted, the unloading phase caches all of that data (despite explicitly using :Destroy()) into untracked memory, and then loads the new map that doesn’t use the same asset as the other one. As such, we have a huge spike in UntrackedMemory that never fully goes down, ever:
We have 8 maps, as such this applies to all of them. This can lead to a huge inflation of the UntrackedMemory. Despite this, we intend to have more maps. Why ? Because this is what Roblox is all about after all. We want to break the grounds, we want to truly bring to the platform what has never been seen before, using all of the tools Roblox provides to us. The platform is going in a direction where experiences are now available for a more mature audience, where those amazing tools are at our disposal to be used, and we can clearly see it by the official funding of Frontline (Game funds project), where the guns, maps, character are all fully PBR, with absolutely mind boggling material variants, effects, sounds, etc.
But if we apply that logic to the full map roster, UntrackedMemory gradually quickly goes over 2.5 ~ 3 Gigabytes, coupled with some memory leaks, Roblox’s own systems, 44 players servers (and we also intend to have more), and the server immediately shuts down after 3 maps have been loaded in. Of course, we could do things differently, obviously there are things that we, as a studio, needs to fix. Memory leaks, offloading some work to the clients, etc, are all improvements that are necessary at this point, but they are absolutely ridiculous compared to the immense data that is stored by the server caching things that we do not want to be cached.
We want the server to destroy the map, not cache it. It is already there anyways, it’s already in server storage, is there a need for caching ? This is hugely limiting our ability to improve the game, and is detrimental to us. There is a loading screen when the server loads the map, so it doesn’t really matter if it takes 10 more seconds for the server to load the map if it doesn’t exist in that black box of a cache, over which we have no control.
Speaking of having control over memory, here are solutions that we discussed with the team over here:
-
Provide us with better memory management tools. Yes, this is straight and brutal, but like said plenty of times, Roblox is headed in a direction to empower creators and innovate; this is simply not possible if we do not have the proper memory management tools that lets us clean the cache or clean certain areas of the cache, or outright prevent caching for certain instance. A new destroy method could be implemented to prevent the aforementioned issue.
-
Increase the max server memory. Like prefaced earlier, 6.25 was very good a few years ago, but this is simply not enough now for games that are truly utilizing all of the capabilities that the engine provides to us. And I am willing to bet that funded projects may face the same issue rather quickly at the pace that they are going.
-
Do not populate the PhysicsPart memory category (which is most likely to be the Physics Octree part of the server, but that we have no certainty since there is no documentation around that) when anything is in ServerStorage.
I feel like this feature request is long enough, and highlights the major drawbacks of how servers currently behave right now.
I would like to conclude again by saying that this platform is going in a direction that is beneficial for everybody, better experiences, huge amount of varied & diverse communities, amazing tools, all of that are being limited by the servers which have been overlooked now for too long. It has become simply impossible to use the new tools to their full extent. This forces creators to make non-beneficial concessions like cutting down on content, and as such is a loss of revenue for both the platform and the creators themselves, and a source of immesurable frustration. We really want to prove that there is much more to Roblox than wacky blocky characters, that full immersion is possible, that triple A games are not just a dream but a real solid possibility. but for that we need improvements on the servers, or there will be no ways for us to use everything at hand other than barebone games or showcases.
