Methods of reducing network and physics lag with large numbers of humanoids (?)

I’m creating a game with NPCs, and by far the major bottleneck to the performance of the NPCs is the roblox-controlled logic behind the replication and physics handling of the humanoids.

My NPC handler is relatively optimised, but even with 100 NPCs in an in-game server, several look like they’re stuttering, even with normal game ping. I am assuming this is because of a bottleneck in replicating all of these humanoids to the client. For comparison, 300 NPCs in Play Solo has around 10% script activity, and generally runs fine on a 2012 Macbook Air (very low-end). This is without any animations yet, which will be handled client-side.

My question is: what additional steps can be taken to reduce the inefficiency of using humanoid objects for NPCs? Specifically in terms of server-client replication.

In an ideal world, for example, I simply wouldn’t even replicate NPCs to clients that are too far away. As far as I know, however, something like this is not possible (there is no way to have this extent of network control).

Things I’ve done:

  • disabled Climbing, Swimming, and Seated humanoid states (positive effect on performance).
  • ensured collisions are not enabled between NPCs via collision groups. (positive effect on performance).
  • optimised my NPC handler to the extent that it is not the limiting factor in performance.
  • using R6 to reduce physically simulated parts + joints
  • setting CastShadow to false for every NPC part, to improve client-side rendering performance (not the issue here)

Any ideas regarding this would be greatly appreciated.

I’ve attempted to make custom NPCs with physics driven by BodyMovers. This turned out to be viable on a relatively flat map, but completely nightmare-ish on a more complicated and dynamic map. To avoid an absurd increase in development time and the complexity of my scripts, plus the performance nightmare that will probably be 300+ NPCs trying to navigate their way around a dynamic map with a boatload of raycasting calculations, I have to compromise and stick to humanoids.

If you want any additional information, my NPCs in essence work by:

  • checking if a direct ray can be cast to the target player, and if so, walk to their position.
  • if not, follow pathfinding until a direct ray can be cast.
  • All NPC models have the network owner set to nil, and are handled entirely on the server.
  • physics is manipulated only by using MoveTo and jumping
9 Likes

I’m having a very similar issue.

One possible solution might be to use TweenService on BodyPositions that are inside HumanoidRootParts for movement.

Shameless game plug:

3 Likes

I tried something like this, but using BodyVelocity and BodyGyro. It was okay, but I found it wouldn’t really work with a complex game map (both in terms of required performance, and because my math ability is somewhat limiting). At that point, your network woes are somewhat alleviated (because you can handle so much more on the client instead), but your development time and the overall reliability of your NPCs always working properly in a complex map–not so much.

I tried to look at luanoid for a bit of an example on how to handle a custom humanoid, but if you look through it, the amount of math required is really insane (it’s particularly crazy considering it functions entirely off of user input, but still).

It just goes to show the crazy amount of work you have to put in to achieve a decently functional custom humanoid in a complex environment. From the creator of luanoid:

Quick caution: Luanoid represents only about a few weeks worth of work between me and LPG. In it’s current state it’s more of a experimental research project, and not a complete character system.

I would personally recommend against using it unless you can afford to invest in the work to “finish” and customize it.

In no way can I see custom humanoids replacing Roblox ones for advanced behaviour, except in instances of 2-dimensional or simple maps.

2 Likes

I just realised something obvious, which is that I’m using R15 (much more parts and joints to simulate) instead of R6 for my NPCs. Switching to R6 has made a decent improvement to performance, and has eliminated physics stuttering it seems for ~100 NPCs

This is still just a band-aid rather than a solution though, because I still want to create relatively complex (i.e. more similar to R15 than R6) NPC models in the future. 100 NPCs is also still not that much.

Humanoids are expensive by nature. If you’ve done everything you can possibly think of without potentially botching functionality or producing unintended results, that’s probably the upper extent of what you can achieve. As it stands, the engine is incapable of handling humanoid NPCs en masse.

In terms of future prospects, the Future of Avatar RDC19 talk may interest you, since it involves discussions about the Humanoid and all. There are no guarantees yet but there is a possibility that a number of features will be ported to Lua or be available for us developers to tweak to our needs.

The link above starts you at the relevant section which is Humanoid Componentisation.

5 Likes

Thanks for the link. My Christmas wish would be specific replication control over humanoids, or instances in general. It would be great if we could somehow control replication on a per-instance basis, for example disabling replication for a certain humanoid to a certain client (mixed with our own client-sided logic). I’m not sure how feasible this idea is, but it’s an idea :slight_smile:

I’m sort of thinking among the lines of using a RemoteEvent to send network data about an NPC’s position to the client (which then animates it properly client-sided), like you could with a custom NPC, but with humanoids.

2 Likes

I think just about any developer would greatly appreciate selective replication for cases beyond NPC handling. I know that it’s popped up as a request before. For now, you have to rely on odd tricks and unsupported hacks to get by as much as possible and to the extent of what the engine will allow.

In terms of NPC replication, you’ll probably want to take what the engine gives you and reduce that work. For example, you can represent the NPC as a single part via the server and have the client do the rest of the visualisations and everything for it. I believe a game currently does this to be able to support a large number of NPCs but I’ve never tested it and I don’t have any data; it’s hearsay from a while ago.

4 Likes

For example, you can represent the NPC as a single part via the server and have the client do the rest of the visualisations and everything for it. I believe a game currently does this to be able to support a large number of NPCs but I’ve never tested it and I don’t have any data; it’s hearsay from a while ago.

This is a good idea, but having humanoids as one part has resulted in a lot of unexpected behaviour for me, like MoveToFinished never firing correctly (although it’s already glitchy as it is…)

I’ve done exactly this with my game Shard Seekers, so it is possible, although the NPC’s are located in the server’s local camera and I replicate the cframes over remote events. NPC’s will also only simulate if there is an observer nearby, making the world much more scalable. My ‘Luanoid’ implementation also allows humanoids/quadrupeds to throttle/sleep if nothing is happening, making idle NPC’s much more efficient. This takes an insane amount of work though. My favorite part was generalizing character physics, so the server and client run the same character code but with a different script connected to the controls.

11 Likes

I haven’t heard of the local camera thing before – I’m assuming anything in the server’s camera won’t be replicated at all? Is this the case even for Humanoids?

The most salient problem to me, though, would be effectively tweening the received NPC locations on the client, so that the movement looks smooth, especially when you have a complex map (eg. the map I have been experimenting with is some randomly generated roblox terrain with a lot of verticality and variation). How do you handle this on the client?

Yep! Although be sure to explicitly set the physics owner to avoid NPC’s freezing because the server tries to change ownership automatically; Figuring that glitch out was very frustrating.

To be fair I’m not sure how well the server would run 300 simulated NPC’s. I would need to reduce the number of foot raycasts on the server, meaning they may have trouble standing on thin parts. The game could have thousands of static NPC’s on the other hand.

3 Likes

The client has an lua cframe controller connected to a lua character graphics object. It just receives cframes along with a delta time from the last cframe, then adds it to a stack. It goes through the stack clearing it while lerping from one cframe to the next, as requested by the character graphics. It does some optimizations like delta encoding and sending Vector3’s when only the positions changes by a certain amount. Clients also request cframes at a certain rate based on how often the value is requested (so far away characters replicate at slower rates.) character graphics/animations also need to be manually replicated and connected to these cframe objects, so it’s quite an undertaking.

What’s cool is that the map gui could replicate player icons along with their cframe controllers, and it can reuse the same controller as the player’s character without there being any connection to the character graphics.

2 Likes

What if there is a scenario where lerping between 2 consecutive positions received on the client causes the NPC to appear to clip through obstacles? This could, of course, be alleviated by increasing the frequency of data, but that brings you back to network bottlenecks with large numbers of NPCs (although surely not as bad as humanoids, considering the potential for more selective networking).

Additionally, the whole server local camera thing seems shady. I have a feeling that some data will still be replicated (my experience with humanoids has been consistently weird behaviour).

I think I’ll play with this idea, considering that I am already going to handle my animations on the client. (If I did it on the server, I think it would catch on fire).

To make the movement more smooth, I think the server could also send the NPC’s velocity, plus it’s target position. My NPCs are always moving, so that’s also another challenge.

P.S. your game is very impressive :slight_smile:

2 Likes

Thanks! :grinning:

It sends at a rate of 1/s only when very far away, and sends nothing if NPC’s are idle. There are some cases where it takes a moment for the higher rate to kick in if you approach an idle NPC before it moves (and thus updating your request rate), but the characters are purely graphical so there wouldn’t be any issues if clipping did occur. CFrame updates on the server use a scheduler that will throttle if it takes to long to execute, so slow updating could occur if there are loads of NPC’s.

A huge perk to a system like this is real-time server-side sanity checks for cframes. The server has info about the clients custom character (like walk speed, flight speed, swim speed, etc.) and can verify that a client isn’t moving too fast, and can simply snap the client back if they try to teleport client-side.

6 Likes