This is a problem inside my game, which actively uses remote events and remote functions, since this is a communication game, the server has a maximum of 110 players, but the server has too much memory in CoreMemory/network/raknet.
1 GB of memory, is this normal? If this is expected, are there better ways to use remote events and remote functions?
This affects the performance and stability of servers.
12:42 timestamp:
16:20 timestamp:
Expected behavior
This memory area should use an adequate amount of memory, at least not more than 300 MB or so, and stay at this point for the entire active operation of the server.
What kind of data are you sending through remote events and functions?
Do you notice CoreMemory/network/raknet consistently growing, or does it plateau around 1 GB of memory?
How long have you been experiencing this issue?
Hi @PlumJar ,
The client sends some data, some numeric and string values, and a couple of small tables to the server. But the server very often sends large tables to the clients in order to keep the client’s data in sync with the server, these tables are very large as they are used by the client when it receives them.
According to my observations, this area of memory grows slowly, at first it may seem that it stands still, but in 2 hours it can swell by 30-80 MB, indefinitely, until the server crashes due to memory overflow.
A full server with 110 players will be enough for about 17-30 hours of continuous work, after which it will be crashed. If you need, I can send a link to my game in a direct message so you can watch it yourself.
Thanks for providing that. Sorry the patch did not fix the issue for you; we will keep investigating and I will let you know when I have another update!
@pozzzy333 Still investigating. Any idea when you first noticed this issue? And does this issue show up in any other memory categories, or just CoreMemory/network/raknet?
I noticed this recently, but it seems to me that it has been problem around for a long time, because I noticed performance problems for a long time, but could not know their cause for a long time. I also have problems with UntrackedMemory, I can’t know exactly how it is related to raknet, but UntrackedMemory can reach 4-5 GB and the server will crash after 50-60 hours of runtime
Thank you for the insight here - my game (link here) has been experiencing a very similar increase (under network/megaReplicationData, network/raknet, and network/replicationCoalescing as well as UntrackedMemory) for the last two years. Servers were crashing at 6.5GB of memory after an hour of playtime. Our solution after a few months of trying to debug it was to increase the server player cap to >600 which gave us 12GB of memory for breathing room.
I have noticed, however, that this memory leak doesn’t seem to be passive - instead it looks to be actively caused by some event on the game. I’ve tried narrowing it down to a particular action but haven’t had any luck. @pozzzy333 have you noticed the same? (the rate of increase isn’t constant).
Hi @unix_system Thank you for sharing your report. Sorry it seems the previous time you reported this issue the report went stale without resolution. The more information I can get, the better - I will keep you guys updated
Yes, the growth rate is not constant, and the server may crash at different times, I noticed this thing when I started getting different memory stats after the same amount of time. Most likely it depends on the actions of the players, perhaps some players perform a certain action more times than at other times, most likely these are remote events and remote functions. My game uses a large number of remote events and each player sends and receives them at different times and frequencies. Perhaps there is often synchronization of requests, when the maximum number of players simultaneously send requests, creating mini DDoS, dynamic systems can synchronize with time for a certain period of time. This is a thing of servers with a large number of players. (This is just my guess and may not be accurate.)
Just wanted to post a quick update before the long weekend — I have been unable to repro using some test places I prepared which exercise things like remote events/functions etc, but of course these test places are less sophisticated than your guys’ live games. This, along with the suggestions that the growth rate is not constant lead me to agree that the leak may be related to some specific behavior. Next week I will be monitoring memory stack traces to hopefully see exactly where this memory is going.
We’re also noticing a similar issue with raknet and untracked memory growing to high levels (although not as pronounced) in our game with 65 player servers;
I’ve also noticed this in my game that has 100 player servers for at least a couple months now. network/raknet is consistently over 1GB on older servers.
Here is a screenshot from a server that is 22 hours old:
I just looked at my old notes where I monitored this issue in November 2021, most likely I met this issue earlier, but I don’t have much information about it, it’s very old
