Cannot selectively replicate/unreplicate model with ModelStreamingMode.PersistentPerPlayer

[Tomarty]

When a player’s ReplicationFocus goes near a model with ModelStreamingMode set to PersistentPerPlayer, the model permanently replicates to them. I have StreamingMinRadius and StreamingTargetRadius set to 64 and StreamOutBehavior set to Opportunistic.

My game uses a custom system where character graphics are communicated over remote events and created client-side. I’ve manually replicated character positions/rotations over RemoteEvents for years, and I recently made the switch to the new model:AddPersistentPlayer/model:RemovePersistentPlayer methods. I was really excited for this.

I managed to hack the behavior I need by not setting player.ReplicationFocus, but there are still issues:

1. Models still do not unreplicate when RemovePersistentPlayer is called. However, if I parent the model to nil then back to the workspace, it seems to unreplicate. This trick does not work when ReplicationFocus is set for the player we are replicating/unreplicating the NPC to.

2. Some players are now crashing consistently. Their RAM usage skyrockets to 8gb+ while the client is frozen. Probably because Replication focus is not set, but I recently started using Actors/multithreading on the client so it could be that. (It seems to happen when lots of characters are on screen and the ragdoll animation starts simulating client-side. For the ragdolls I do lots of actor:SendMessage calls to set up water simulation for the parts.)
   

   
   
   One of the users reporting this issue says they have dump files.

3. Players are experiencing graphical glitches with client-created smooth terrain used for water and grass details. I assume this is because ReplicationFocus is not set. It could be due to StreamOutBehavior being set to Opportunistic though.
   

   

I should probably clarify that my entire map is replicated manually over remote events and is constructed client-side with my own streaming system. These NPC models are the only parts in my entire game that are replicated using StreamingEnabled. The server has a physics-only version of the map initialized within a Camera instance.

Expected behavior

Ideally the PersistentPerPlayer should not replicate to the client unless model:AddPersistentPlayer is called, and model:RemovePersistentPlayer should cause it to unreplicate. If this is not the intended behavior, I would appreciate an additional ModelStreamingMode with this behavior. It’s just more stable and gives low level control. It’s a waste of server and client resources to have these parts streaming when they don’t need to be. There are valid use cases for manual model streaming like this, for example private player homes or special rewards that appear for specific players.

Even so, the model should un-replicate when it leaves StreamingMaxRadius, so long as the player is not in the persistent player list and StreamOutBehavior is set to Opportunistic.

[boxheadedperson]

Hello @Tomarty,

1. PersistentPerPlayer models behave like Atomic models when interacting with a player that is not in the persistent list. So if the model is within the streaming radius it will still stream in. Your observation regarding un-replicating (streaming out) the model when it is outside the max radius is correct. I tried to reproduce this locally and this is the behavior I saw once the player is removed from the persistent list. If you are experiencing different behavior and have a simple repro I’d love to take a look.

2. Do you know when the crashes started? Are you able to reproduce the hang, or see the large RAM usage locally? If so I wonder if the memory categories under Developer Console could shed some light on whats happening.

3. We have seen similar terrain glitches before when the minimum and target radius are the same but we had thought we had fixed it. It might be worth trying to set the radii to separate values and see if that resolves it. Still, this is an issue we want to understand and fix. I’ll try to repro myself but if you (or your players) have consistent repro steps I would appreciate it!

[Tomarty]

@boxheadedperson

PersistentPerPlayer models behave like Atomic models when interacting with a player that is not in the persistent list. So if the model is within the streaming radius it will still stream in. Your observation regarding un-replicating (streaming out) the model when it is outside the max radius is correct. I tried to reproduce this locally and this is the behavior I saw once the player is removed from the persistent list. If you are experiencing different behavior and have a simple repro I’d love to take a look.

For my use case it would be best for models to never replicate unless AddPersistentPlayer is called, and promptly stream out when RemovePersistentPlayer is called. A developer might not want to replicate specific user creations for privacy reasons, as an example. An additional ModelStreamingMode would be ideal. It would more directly solve this problem.

What’s happening is the player goes near these NPC parts (used for solely position/rotation/velocity replication. Even if ReplicationFocus is moved far away and the player is removed from the persistent player list, the model and its parts remain within the client’s workspace.

My usage is somewhat complicated. I have ReplicationFocus set to an anchored 1x1x1 part, parented to a Camera instance in the server’s workspace (it’s parented here because I don’t want these to replicate to other players for privacy and scalability reasons.) This part follows the location of the custom character the game uses.

You can see why I was replicating character position/rotation over remote events. I’m adding NPCs, and manual replication just doesn’t scale well and isn’t as responsive. I’m trying to implement a reliable solution to this problem that uses Roblox’s built-in physics streaming. I already do spatial partitioning to know what to replicate, and have logic that controls the maximum number of characters that will replicate to prevent lag. Maybe with more testing the model might unload, but it didn’t seem to when I tested it. I just need to be able to selectively replicate the parts so I can attach character graphics to them client-side.

@boxheadedperson

Do you know when the crashes started? Are you able to reproduce the hang, or see the large RAM usage locally? If so I wonder if the memory categories under Developer Console could shed some light on whats happening.

Someone said they dove into the water when the crash happened. I told them to look at the memory category, but they said that the memory only started going up after it froze.

My terrain’s collision group is set so that it doesn’t collide and I just create the water’s surface, but I haven’t had issues until after turning StreamingEnabled on.

@boxheadedperson

We have seen similar terrain glitches before when the minimum and target radius are the same but we had thought we had fixed it. It might be worth trying to set the radii to separate values and see if that resolves it. Still, this is an issue we want to understand and fix. I’ll try to repro myself but if you (or your players) have consistent repro steps I would appreciate it!

The terrain isn’t being streamed in. The server has no voxels. I’m adding water and patches of grass client-side for graphical effects.

---

The server’s workspace looks like this:

The parts in the NPCs folder are the root parts of the NPC physics model. The rest of the NPC’s parts are in the server’s camera because the client does not need them, and they contain attachments/forces that I certainly don’t need to replicate.

The client’s workspace looks like this:

In this example the client has moved far from the NPC models and two remain replicated. On the server, the ReplicationFocus is in the correct location, and GetPersistentPlayers confirms that these models should not be replicated to the player.

[boxheadedperson]

I will bring your idea about a new ModelStreamingMode to my team and I’ll get back to you after we discuss it. From your screenshots, I would expect those 2 models to be garbage collected given they are far enough from the ReplicationFocus. Locally I created a PersistentPerPlayer model outside of a players target radius and calling AddPersistentPlayer and RemovePersistentPlayer streamed the model in and out. My test was using the player’s character location to stream so I can try again with ReplicationFocus instead. In your experience, is the custom character the player controls under their Player object?
Also, when do you call Add and RemovePersistentPlayer?

I will look for recent crashes from your experience and see if I find anything interesting. I cant think of anything streaming related that would cause this (the hang/freezing or memory usage).

If the water and grass is client-created then streaming should not stream it out so this might be a rendering bug, I will ping someone from that team to chime in about how to debug this separate issue.

Since

[Tomarty]

@boxheadedperson

I will bring your idea about a new ModelStreamingMode to my team and I’ll get back to you after we discuss it.

It would be nice if this could be utilized without StreamingEnabled being set to true. The model will simply not replicate to clients unless specifically added for a player. I disabled StreamingEnabled many years ago because I had so ongoing stability issues with my game’s custom characters, and other issues that have since been fixed.

It would be nice to be able to manually replicate just the BasePart instance, but replicating just one part is probably too niche a use case, unless it can be generalized for other types of instances located anywhere in the DataModel. I manually replicate stuff to the client by parenting it to their PlayerGui, sending a reference over a RemoteEvent, then parenting it to nil; This setup works for me though, and a few dozen unnecessary model instances in the client’s memory as a means to replicate parts is fine.

An alternative to a new ModelStreamingMode could be to allow setting StreamMinRadius and StreamMaxRadius both to 0, in which case it would just not replicate anything except for PersistentPerPlayer models added explicitly. If there are technical limitations for it being set to 64, it could clamp it at runtime or specifically allow setting to 0 in studio.

---

I call Add and RemovePersistentPlayer based on approximate distance. The server has a spatial hash for characters, and it connects the players as observers. I use multiple sizes of spatial hash so that different sizes of details stream effectively.

Even when I stop using Add/RemovePersistentPlayer, and cause it to replicate by going near, it still sticks around. The behavior seems inconsistent. Sometimes specific models clean up, and sometime specific models seem to stay replicated indefinitely.

---

The crashing and water/grass issues only started happening after the update I pushed out yesterday. The only changes I made that would stress the engine are:

• StreamingEnabled set to true (so that the NPC models can be replicated manually.)
• I started using Actors and actor:SendMessage to multithread my custom character controller, as well as to multithread custom water AABB buoyancy calculations for ragdoll parts that are created locally. Up to 100 SendMessage calls can be done at once when the ragdoll parts are set up, depending on the complexity of the character.
• This update made the dragon character much more accessible. These use 100+ bones and lots of 1024x1024 PBR textures. This update added 32 1024x1024 PBR textures as part of the ice dragon cosmetic pack.

I suspect the graphical issue is related to StreamingEnabled. I’ve been setting terrain on the client for years, and only had issues after this update. Unless it’s a very strange memory issue: a symptom related to the crash.

The crash always ends the same way, with the client freezing and memory usage skyrocketing. Two players reported it occurring on their Windows laptop, seemingly in relation to some physical interaction with their character and there being lots of characters loading nearby using the “Tumble” animation, which causes a ragdoll to simulate on the clients of people who have that character on-screen. Turning the camera can cause lots of characters to load all at once in a crowded area because the graphics are loaded client-side, which might stress the client. I’ve probably received <10 reports related to the crash, but most don’t provide much detail.

One of the players who reported the issue claims they dug through the dump file but found nothing of interest. The issue happens suddenly with no way to open the developer console or anything.

[lolmanurfunny (lolman)]

@Tomarty

It’s just more stable and gives low level control. It’s a waste of server and client resources to have these parts streaming when they don’t need to be. There are valid use cases for manual model streaming like this, for example private player homes or special rewards that appear for specific players.

I’ve been in need of such a feature for the longest. Streaming doesn’t respect developers’ intentions in its current state, it’s too rigid.

@lolmanurfunny

Everyone, not just fellas at roblox must understand that a developer’s intended replication behavior for instances in their game is contextual to what they desire and thus the context they desire often goes far beyond distance

[Tomarty]

The freeze happened to me for the first time today. I had the window minimized when it happened.

It seems to correlate with the tumble/ragdoll animation. It uses actor:SendMessage to instruct a actors to simulate water buoyancy on each part.

It might be because I was using BodyForce (it was easier to get working and didn’t require an Attachment.) I’ll switch to VectorForce and see if it helps.

So far it seems the crashing issue was caused by using BodyForce instances.

[boxheadedperson]

The feature request is on my team’s radar and we have begun discussing this. When we have reached a decision we will let you know. We are aware this is a pain point for you and other developers so we will try our best to help.

---

I cannot seem to reproduce the inconsistent un-replication, I tried with Add/Remove APIs and just automatic streaming.

persistent per player automatic streaming

persistent per player add remove

Would you mind sharing your place file / experience so I can try to repro? If so, please dm it to me and I will be very careful with it and delete it once I am no longer using it. I was also able to find a crash dump for a hang that occurs in Streaming code. I’m not sure how this hang could occur so having access to your experience would also help me investigate this.