Roblox internal memory leak causes my game servers to undiagnosably crash every 3 hours

Reproduction Steps
I have spent the last few months trying to narrow down the problem to a script in our game without much luck; we have rewritten about 50% of our codebase in search of this issue and to no result.

SCI - Pathos III - Roblox - The issue occurs here in our live game, where servers reach between 6GB before crashing, and occasionally get the cap increased to 12GB.

Expected Behavior
Servers should either not crash, or if there is a script-caused memory leak that memory leak should be attributed to a specific script label in the developer console.

Actual Behavior
13 minutes into game server start:

Our in-game graphing for server memory and nil instance count over time
Vt7sFUt.gif (1154×436) (imgur.com)

Screenshot of Untracked Memory

PlayFab statistics for server lifetime, with Server Memory on the Y axis and time in the X. The key is each JobId.

The obvious pattern here is, according to some unknown factor (we have tested against player count, joins, deaths, respawns, vehicle spawns, streaming loads, weaponry, remote traffic and all seem to have no correlation to the rate of increase) the server memory sharply increases to some point, and then either stops for a while or continues up eventually hitting the ~6.25GB limit for Roblox servers before they crash.

Screenshot of our server memory

How I calculate Nil Instances
local function getNilInstances()
	return stats().InstanceCount - #game:GetDescendants()
end

This gets the number of game descendant objects and compares it to the stats() instance count, to identify the number of non-existent children.

The memory is distributed, apparently randomly between Untracked (implying a memory leak somewhere on our end), network/megaReplicationData (which I have been unable to find any information on), internal/RCCService, network/replicator and network/raknet

Memory Profile of game:
memProfStorage44428.json (5.6 KB)
d
Our game has approximately 100,000 parts (fluctuates due to building needs) - this is obviously very large, however if this was to blame I’d expect the memory to stay at a constant level rather than constantly increasing.

Issue Area: Engine
Issue Type: Crashing
Impact: Very High
Frequency: Constantly
Date First Experienced: 2021-05-01 00:05:00 (+01:00)
Date Last Experienced: 2021-09-10 00:09:00 (+01:00)
A private message is associated with this bug report

27 Likes

I would like to provide additional information

By looking through the forums i managed to solve the issue of ‘nil instances’ increasing over time by calling game.Debris:AddItem(Player, Time) connected to the PlayerRemoving event, but unfortunately did nothing to solve the memory steadily increasing.

the fact that your nil instances never scales down overtime on average is also a bad sign,
this could indicate your code is holding reference to that Instance object in memory, or when calculating with your function to get nil instances count, that the engine internally causes this leak to track instances that should’ve been garbage collected but can’t because those instances are now being referenced and the collector will not touch the instances increasing memory overtime.
This is pure speculation and I do not understand anything internally in Roblox but if someone can get an idea off of this and go further that is very much appreciated.

2 Likes

Thanks for the report! We’ve filed a ticket to our internal database and we’ll follow up when we have an update for you.

3 Likes

Some additional information I would like to provide

I created a test place that acts as a stress test for memory by having a script create 100k parts inside of lighting every ~25 seconds, followed by clearing lighting a few seconds later, all in a loop, and this was the results i got:


Memory stats sourced from custom made memory tracker that is also located in the place

As seen in the graph, the total memory almost never went down while on the other hand, the untracked memory accumulated with almost every loop.

The place: BUG REPORTING - Roblox

2 Likes

Seems suspiciously similar to what’s been ruining one of my games?

Memory rises initially as things are loaded up on the server, then plateaus between sudden large jumps.
There’s no correlation between player count or any in-game event that I could find, but haven’t gone as in-depth as you. I just can’t figure out how I can possibly be triggering sudden gb spikes in server memory within the span of a minute.

2 Likes

Yep this is pretty much the exact profile I get too. The closest I could tie it to is player respawns, however I haven’t been able to narrow it down past that. Part of my problem was obscured by a now fixed HumanoidDescription memory leak.

Have you tried plotting respawns against server memory?

1 Like

It’s funny you should say that actually, because I asked @Orlando777 to try and find any clues as to what was causing the memory leak, and he said it happens when people reset. Thing is I couldn’t replicate it and there just was very little code going on with character death and loading which was easily reviewed and ruled out, so I wrote it off as coincidence.

I’ll put in some code to plot respawns and memory when I have some time and report back.

4 Likes

Done some plotting long term plotting of the various memory items against up time for one of my places that has been mainly affected by this.

Max Players: 50

In all three instances, there is one or more sudden spikes in Core Memory that climbs uncontrollably out of control and for no explainable reason, increasing by several hundred mb before sharply falling to roughly its previous level, with the total and untracked memory retaining some of that amount, never falling.

Having been fortunate enough to be in the first server when the memory spike began, I was able to determine that the increase is almost solely caused by network/replicator:

It is odd that it increases so rapidly despite everything else non-core related remaining the same, instances especially.

1 Like