Your playerbase being established helped with that. My player base is only in the hundreds, and it can be pretty demotivating from players to join a game when there is no servers being listed in the web browser, especially for a desktop only game.
14 Likes
It really isn’t
I too had to start from somewhere.
A playerbase of hundreds is plenty that a lobby system would work without fail. What really matters is what player count they see when clicking on your game. Hell even Naval 1945, one of my other games with a concurrent playerbase of dozens, still gets sizable amounts of players, with or without the lobby.
12 Likes
I will keep that in mind. However, like i previously mentioned, this is the kind of effort that I don’t really have enough time to put into. On top of that we shouldn’t really have to rely on such hackiness in the first place. I’m not really sure if I want to set aside the simplicity of just clicking on a button to play the game in favor of making a lobby for more server memory, and the potential chance of losing a sizeable portion of the playerbase even if you guarantee me that this won’t happen.
All of the issues I originally brought up in the feature request just should not happen. The octree being loaded with maps in server storage, really makes no sense whatsoever to me. Sure you have to keep the instances somewhere, but their collision data and partitioning do not have to be physically loaded into the memory.
This feature request would literally benefit the entire platform, both games that are already live and future games, especially the upcoming PBR projects that are being funded by Roblox themselves.
10 Likes
I am surprised the memory isn’t at least 12 GB. RAM isn’t that expensive these days either.
14 Likes
Considering the average server has at least 32-64 GB of RAM, yes, it is not expensive.
16 Likes
I believe I am experiencing the same thing in my game that cycles between terrain maps. Check the topic I made here and reply with your case if you can: Terrain.PasteRegion memory leaks
I am convinced there is some kind of memory leak when loading/unloading terrain regions. Clearing the terrain doesn’t clear the untrackedmemory (why is it even untracked??). I had to implement an auto restart after only 8 hours because servers would reach their memory limit before then due to the map cycling. It’s really nonsensical that so much stuff gets thrown into the ‘UntrackedMemory’ category with no way to even determine what it could be.
9 Likes
Interesting !! I think this feature request should turn into a bug report of its own. I’m pretty sure it will get even more traction there than here. For some reasons it feels like the Roblox Staff isn’t reading a whole lot in this devforum category…
6 Likes
It would be nice if Roblox would do a hard GC sweep before just shutting the server down too as it nears the limit. The incremental sweep model is nice for performance but I’d take a 50ms freeze over a server crashing any day.
One thing I’ve noticed is that if your game actually just needs a lot of memory is that it can take a very long time for GC to actually crawl through the whole heap, which is problematic since that means it might take 2+ minutes for unreferenced data to be collected and cleaned up. A lot of waste occurs in that time. This can also make you think you have a memory leak when you actually don’t, you just need to wait around for 2 minutes before the memory is deallocated. So perhaps some way to signal to the GC that we’re done with some data and have it force clean up on the next step might make sense. I think Roblox is really trying to go for an “it just works” approach, but I think there are always going to be edge cases where it doesn’t.
One case I’ve observed, and this could 100% be optimized, is where we have this really old script that has a Heartbeat connection which calls Players:GetPlayers() every step, so ~60 times per second, and we have 55 player servers, so it holds on to 55 player references 60 times per second for 2 minutes, like 396k references to players before GC happens. This only consumes a few MB of RAM but it appears as though it might be a memory leak when it, in fact, is not. When you think about all of the different loops allocating data that don’t actually keep any references past that single frame it can really add up.
Obviously you don’t want to just clean up all references every frame, but some sort of way to prioritize the GC might be helpful here, combined with a full heap sweep before killing a server and I’d imagine most games that don’t have actual memory leaks might be able to survive on 6.25GB of RAM.
14 Likes
Interesting. Does setting it to nil in the scope of the loop change anything?
4 Likes
No, setting something to nil doesn’t deallocate the memory, it just removes a reference to allocated memory, the garbage collection will eventually crawl over the portion of the heap which contains that allocated memory and then deallocate it, provided there are no strong references to it remaining.
5 Likes
Try and have a lookup table in which you add/remove players as they enter/exit the game or during the relevant events in your game. This will help.
Like said in the feature request for me the highest impact is made by maps being in server storage and fully loaded into the physics octree and loaded into the untracked data as it is being cached when removed.
The kind of semi-micro optimization that you’re taking about I kind of have ruled out over the whole code. Sure there are some mem leak as the servers ages up but it’s nowhere near as bad as it used to be before the aforementioned problem arose and I had to tackle almost every memory leak before releasing the update.
Either 6.25 gigs is not enough, or we need better tools to make sure GC happens when a map unloads, or Roblox needs to fix the caching/octree issue.
4 Likes
I wasn’t implying that my experience was the chief issue, or that it was even an issue at all. My whole purpose with my statement was to point to the fact that having tooling to prioritize deallocation of memory would be ideal, and such a case was one of many possible scenarios where it presented itself. What I’m essentially saying is that as the heap grows, the problem that I’m talking about also grows, because it takes GC longer to crawl the heap. Why is that a problem? Because these sorts of “memory growths” are extremely common, and they add up to several hundred MB of data, and unless you’re aware of what I just explained then you’re most likely going to think this is a memory leak, as I did. Granted, it’s all more about being educated in how GC works than it is about the “issue” itself.
This is a feature request, not a request for help. I’m simply sharing my experience with memory use. I’m quite aware of how simple the problem I explained is to fix, I just thought it was a noteworthy experience that other developers might also benefit from hearing about.
If you read what I said, I literally agreed with you that we need better tools to more aggressively deallocate memory or we need more memory. 
3 Likes
Sorry, I completely misunderstood what you said.
You’re right, what you provided as a personnal experience is a good example of how easy it is to grow the heap and how long it might take for the GC to collect and free all of that memory that can quickly grow to hundreds of MBs while also foolishly thinking it may be a mem leak when it’s actually an intended behavior.
In the long run you start figuring out that what you’re seeing is not memory leak but you kind of have to have some experience in GCing and caching to realize that.
Apologies about the misunderstanding !
4 Likes
Have you considered to use multiple places within one experience? Make a place per map, one lobby place and use that to teleport people around?
I believe we are having an EXTREMELY similar situation. Our game upon starting will duplicate the entire map (a massive 25x25 baseplates) into ServerStorage, to allow future map regeneration. Our servers will max out on memory and crash after roughly 1-2 hours of uptime without StreamingEnabled, and roughly 15 minutes to an hour with StreamingEnabled. Curious to hear if you have StreamingEnabled or not.
2 Likes
Hi ! Sorry I don’t know how I completely missed your reply.
I have it disabled despite multiple attempts to enable it for my game. I feel like we lack too much control as of yet with Streaming for it to be viable. It has always been causing a bunch of issues when trying to have it enabled and as a result isn’t really worth the effort, I can’t remember specific examples as to why it’s not viable for our use case but I know that after spending a lot of time toward having it working, I came to that conclusion.
Glad to hear that having streaming doesn’t help either, at least I’m not missing on something.
This issue is infuriating me, it forces me and my team to pivot to new kind of games because we are completely stuck right now.
The server should NOT cache the maps that are unloaded, and should NOT load the physics octree when all the assets are in server storage at all ! Please Roblox, give us better memory management tools so that I can at least empty the cache after each map unloads.
3 Likes
Yeah, the added memory (1.6 GB for us) that streaming enabled uses actually makes the problem worse. We’ve resorted to just disabling the map regeneration as server uptime wasn’t even long enough for the map to ever regen. UntrackedMemory spikes as high as 1.5 GB after a full map regen (does not go back down, leads me to believe its caching!), and just cloning it into ServerStorage eats up 600 MB by itself. Huge problem for our game as it completely blocks all future map-related updates. Super unfortunate as besides the memory problems Streaming Enabled helps performance GREATLY compared to our own rendering system. Hopes to better memory management!
4 Likes
Yeah that sounds like the same issue. Roblox is caching terrain and parts for maps when it’s obvious that caching should not happen for such large assemblies. We literally do not care if it takes more time to load because it’s not cached, that’s the whole point of having a loading screen. Caching size isn’t an issue on standalone games, but since Roblox is limiting us with their memory size and non-existing memory management tools, it’s shooting itself in the foot, and us at the same time.
Moreover, we can’t even load terrain on client, so there is absolutely no workaround other than copying the BaseParts of the maps themselves client-side, but that introduces a bunch of complexity because now the server doesn’t have access to all of them which means delegating a bunch of other stuff to client as well which introduces security issues and adds more load on lower ended devices…
Seriously we need something quick, I had to abandon the idea of any kind of maps updates for my game as well, and I’m moving on to another project solely because of this very restrictive and impactful deprecated limitation.
2 Likes
As far as I know, the caching is lazy allocation so it gets freed up only as you reach the limit. They don’t particularly bother about having “free” memory on Roblox until it’s needed, same reason why if you switch devices your client will store as many textures into memory as possible.
I find this approach combined with no developer freedom sucks at game scale, and creators have no way to handle device and server crashes from loading in too much data at once.
We run a game with high-res retro building style so we clock over the 100k part count and have had concerns with these limitations before. In our case, StreamingEnabled actually just made our overall server usage worse, without providing us much freedom in controlling what gets loaded/deloaded. We only turned it on because if we don’t then our game isn’t playable on mobile because the devices crash.
There are other kind of memory leaks you need to look out for also, just as an example, if you don’t call player.Character:Destroy() and player.Character = nil before respawning a character it will cause a memory leak. And it’s impossible to have a weak reference to an instance, even if the container is weak, so they need to be freed up properly.
Annoyingly these issues happen to be arbitrarily nuanced because you’re never going to fully know the extent of how your game is handled by Roblox, because the profiler is vague and they won’t give us any documentation.
Haven’t played your game but going to have a guess at some possible fixes.
Consider never destroying maps, and setting aside a tag for parts of the map which need to regenerate. And if they get damaged then mark that, and at the end of a game just regenerate the damaged sections. You would not need to clone models (unless they are movers) till they get damaged.
Otherwise, have you considered encoding your map into a custom rbxm/json format? Then further compressing it by group matching duplicate meshparts, models, and unions. Would remove the data from ServerStorage entirely, and allow you to store it even smaller than if it were a model. And it would allow you to load it dynamically.
3 Likes
I second this. Our game wants to expand in many different ways but can’t because we are limited to the 6.25 GB that ROBLOX allocates to our game.
2 Likes