Server network/raknet Memory Leak?

Yeah, it’s super unfortunate. I hope ROBLOX addresses this in some fashion somehow.

1 Like

Hey guys! Good news is that development on a tool which will help us diagnose this (and other similar leaks) is now development-complete. We are testing internally and I will be using this tool soon to make progress on this report. Thank you so much for your patience, it was important for us to take the long solution so we are better equipped in the future.

14 Likes

Could you elaborate on the server browser you used to artificially cap the player count? I find this very interesting.

Basically what he is saying is that you’d need to implement your own a player cap.

You can do this with a basic implementation which includes setting a maximum player cap in the game, if that is exceeded you can kick new joining players. This could be frustrating to them, though.

The approach I would take is as he stated, using a server browser. Create a “main” place which acts as a lobby and perhaps a gui which lists all available servers and there corresponding player counts. This isn’t an easy approach though. You’d need some good experience to have a top notch implementation.

I’d love to feed you the code, but you wouldn’t learn just by me pasting snippets, but I’ll give you a breakdown of how I’d go about it.

Queue System:

  1. Create a new Roblox place that acts as a “lobby” where players wait in a queue to join the main game.
  2. When players join the lobby, add them to a queue data structure (e.g., a table) and display their position in the queue.
  3. Periodically check the queue and move players to the main game when there are available slots.

Server Browser:

  1. Create a GUI for the server browser that players can access from the main menu of your game.
  2. Use a UI library like Roblox’s UI or Roact to design the server browser interface.
  3. Retrieve information about game servers, such as server name, player count, and other relevant details.

Communicate Between Lobby and Game Servers:

  1. Implement a system for the lobby to communicate with the game servers. You can use Remote Functions or Remote Events to achieve this.
  2. When a player reaches the front of the queue and there’s an available slot in a game server, send them the necessary information to join that server.

Main Game Server:

  1. Modify the game server to handle a limited number of players (e.g., 50 player cap).
  2. Make sure to track player disconnections and free up slots for new players from the queue.

Server Information:

  1. Populate the server browser with information about available servers. You can periodically update this information.
  2. Display the server name, current player count, and an option for players to join.

Joining Servers:

  1. When a player selects a server from the browser, they should be sent to the lobby if the server is full or directly to the game server if there’s an available slot.

Handling Disconnects:

  1. Handle player disconnections gracefully, both in the lobby and in the game server, to maintain an accurate queue and server browser information.

User Experience:

  1. Ensure that players in the lobby have a good user experience while waiting, perhaps by providing them with interactive activities or mini-games.
  2. Keep players informed about their position in the queue and estimated wait time.

Wow that is a very detailed response, thank you. Very helpful, I’ll see if it can be implemented in one of my old games.

Don’t forget to set the maximum visitor count to a high number to gain the memory “upgrade”, set it to like 700 by default.

Sorry I couldn’t provide code to help further,
but hope it helps. Tried to keep the explanation minimal so you’d pick up on the concept easier.

If you have any queries feel free to converse with me via DMs.

Happy coding!

I am getting this, but even worse is the amount of untracked/voxel memory loading/unloading terrain causes with PasteRegion. There just seems like an insane amount of memory leaking on the Roblox engine and not being able to go above 6GB RAM on a 64-player high-action server is obnoxious. I’m forced to reset servers every 9 hours due to these issues and even then it still crashes.

We released our artificial playercap & custom server-list 5 days ago now. It’s been a joy. No more server crashes, NONE!

Once again having servers lasting multiple days. :sunglasses:

image

Hey guys, wanted to post an update. Using the new internal tool, I have found a significant memory leak in RakNet. I am working on a patch now ~ I am hopeful about this one but don’t want to overpromise. Thank you guys for your patience!

19 Likes

Would we ever have access to some sort of tool like the one you are referring to like the performance tab, or task scheduler tab in studio were we can also measure our script performance as sitting inside of a game with F9 open or microprofiler sometimes doesn’t cut it. If not that is ok, keep us updated on any good news to this topic!

The specific tool I was referring to will be internal-only due to its nature- it gives us insight into memory that we manage in the engine and its systems. However, we do want to give as much visibility as possible to you all and the team which owns these memory projects is working on that (I have made them aware of this thread). I do want to point out the script performance tool in studio that currently exists (check out the view tab: View Tab | Documentation - Roblox Creator Hub)

1 Like

I also just now enabled the patch I mentioned last week. New servers should have this improvement. The memory leak I found had a similar nature as described by many of you in this post:

  • in memory/raknet
  • more severe in larger servers with more players
  • randomly spikes memory usage sharply, but never returns to baseline

I tested thoroughly to make sure that this leak was now gone, but of course memory leaks can be tricky; there could be more than one.

Please let me know if you notice improvements, or if you don’t – thank you again for all of your feedback and for hanging with me!

9 Likes

After 16 hours of uptime network/raknet was 30 MB on a full server. I see significant improvement in this area of memory! I think we need to get more messages from people who have experienced the same problem so we can confirm the fix.

1 Like

That’s very exciting news! I’ve been eagerly anticipating replies :smile:

Hopefully @unix_system , @InfiniteEffect , @OutlookG , and others see similar results :crossed_fingers:

3 Likes

Apologies to add onto this thread, but maybe perhaps this is the cause for some players not being able to connect to my game (error 277)? My situation is fairly similar. Server loads in server list pages (based on client sorting preferences) from a offsite web server, and sends the pages back to the client. Roblox server caches those for 5 seconds in case anyone else requests with the same sorting. Requests to that web server usually take like 40-100ms.

Server memory is always below ~1,500, client isn’t very high either. Recv is always in the green so that seems fine.

I am seeing spikes in Total Data KB/s though of upwards of 40,000+, maybe you guys have some advice on how to address that? This is tricky because most servers seem to be fine, yet there are still players who cannot join some for whatever reason.

Seeing a very solid reduction in memory usage for servers in 1% win obby!

2 Likes

Hey Jason! Hmm, that sounds like it may be a separate/unrelated issue – in order for you to get in touch with an engineer who can take a look at that, do you mind please posting it in a new thread? It helps us to track new issues to completion. Thank you for reporting!

1 Like

Hi Plum!

Alright yeah I will! Unfortunately I’m not able to post new threads. I think that feature is disabled for me, is there a way you could enable it?

Hey Jason, I am unable to enable for you ~~ but if you keep checking into the forums, you should be able to post in no time ! (it is enabled based on your activity)

I am seeing significant memory issues with network/raknet in my game. This specific server only had an uptime of about 13 hours when I took this screenshot.

I’m not sure if this is related to the issue, but I am sending UnreliableRemoteEvents 22 times per second for a subset of users who own a game pass that allows them to complete an action in the game very fast.

You can also see that there are large drops in server memory usage when I update the game & migrate/shutdown servers.

1 Like