I’ve been working on trying to figure out what’s causing extremely high memory usage in both untracked memory (3,000MB+) and touchReplication usage (2,000MB+) in our game for a few days now only to come up with nothing. After an hour or so, a Sandbox server (can be found by clicking servers in our game) will crash presumably due to memory.
Additionally, total data sent is also spiking sometimes up to 10,000 KB/s+ which may or may not be related.
Hi, we have some ongoing work to reduce the network and memory usage for touched event. In the meantime, I would try these workarounds if the issue affects gameplay.
After 2 weeks of trying to get to the bottom of why my servers were constantly crashing on my game, I think I also pinpointed the issue to UntrackedMemory and touchReplication having high MB size. Here is a screenshot of the server memory right before the server crashed:
(Place Id = 12081110790, Universe Id = 11727501024, Time: 18:23 UTC, May 9th)
As shown, the touchReplication memory jumps from 1.8GB to 4.8GB within 30 seconds.
Could we have some more info about this issue? Is it possibly related to the cloning of models? Because in my game carts are spawned very frequently and then destroyed once the player stops using them.
I was thinking of reusing these cart models instead of destroying, then cloning them. Would that be worth pursuing? Again, some info on this issue would be helpful. This issue is really hurting my game’s user experience due to servers crashing and data loss and has been for the last few weeks I believe.
In Robloxian High School it seems to have stabilized around 1.5gb in a half day old server. There is no memory being used up by touchReplication though.
The touched event consumes a lot of networking and memory resources because touched events are replicated from one client to server and then to other clients if it triggered locally; otherwise from server to all clients if triggered on server. At the same time we need to debounce the replication which requires a cache. We are actively working on a solution to resolve the issue.
At the same time, if you would like short term workaround, please continue:
I would avoid destroy and clone again if the model is reusable. It definitely benefits in all aspects (not only limited to touched event performance).
If you aren’t using touch events, set CanTouch = false
If CanCollide = false, set CanQuery = false when you have no intent to do any spatial queries against those parts.
Make sure only create touched event listener when you need it. If you have a cart with a hundred parts listen to touched events and then process them in the same way, it probably better to create one invisible big part that listen to touched event and process that single event on the hundred parts.
For untracked memory, we have another team look into it. I’ll share the result once they have it.
Thanks for the response. I actually moved some important touch event listeners to the client and this seems to have resolved the issue now. I also made use of Workspace.TouchesUseCollisionGroups and CanTouch to make the touch event more efficient on the client.
I think specifically there was a touch event (listened to by the server) of a part over which the players have network ownership that I believe was causing most of the lag. Just mentioning this in case it helps your investigation into the issue.
Apologies for updating an ancient thread. We have been revamping our memory system and are working on addressing untracked memory. Untracked memory specifically occurs when memory is allocated outside of our usual memory allocation routines. We have been hard at work improving our memory systems to improve efficiency and properly categorize untracked memory.
The bad news is that this is an extremely large project, and we do not yet currently have an estimated time to completion or concrete milestones to share. Please stay tuned and keep a look-out for any updates to our Memory Tracking system.
For now, due to the size of this work, I will close this as “By Design”.
