Physics discrepancies when performance is stressed

[Jamie_Fristrom]

• Describe the bug. Describe what is happening when the bug occurs. Describe what you would normally expect to occur.

For physics being performed on the client (eg, vehicle driving) when the client is under a physics load of more than around 10ms the physics behavior changes noticeably - physics slows down and some movers appear to be operating at a different relative speed than others and/or hardcoded velocity. The framerate itself can still be fine (~30fps) according to the performance stats. On weaker machines this problem can ruin the playability of a game.
This occurs with filtering enabled and PGS on.

• How often does the bug happen (Everytime/sometimes/rarely)? What are the steps that reproduce the bug? Please list them in very high detail. Provide simple example places that exhibit the bug and provide description of what you believe should be the behavior.

Everytime. I’ve provided a repro test place, which I made simply by taking the driving start place and adding a client-side script that creates unanchored parts to stress the physics and turning filtering enabled on.
To see the bug open this sample place:

Physics Inconsistency Repro Case

In the default mode, the car drives normally, but when you click the Tax Client Physics button I’ve provided in the upper left and then attempt to drive, the car moves forward very slowly and is very difficult to control. (Note - you can use the text box to change the amount of physics stress - performance seems to degrade fairly proportionally to me.)

• Where does the bug happen (www, gametest, etc) Is it level-specific? Is it game specific? Please post a link to the place that exhibits the issue.
  The bug happens to a small extent on every level of [PET WOLF RIDER!] [ALPHA!] Legend of You - Roblox when run on a i5-3210M CPU @ 2.5Ghz/2.5Ghz and 4Gb ram

• Would a screenshot or video help describe it to someone? If so, post one.

Level without physics being stressed:
https://gyazo.com/c68a48c6b1a25db672fd7c23a7498673

Level with physics being stressed - note physics is ~15 ms and framerate ~30 ms. Not unreasonable IMO so it’s weird to me that behavior has changed so dramatically:
https://gyazo.com/569abc65d9b501faf665312a2c830cac

• For graphics bugs, it is sometimes helpful to know your system specs, especially graphics card.
  Those videos were taken on an i7-6700 @ 3.40Ghz/3.40Ghz with 8Gb ram

• When did the bug start happening? If we can tie it to a specific release that helps us figure out what we broke.
  As long as I’ve been working with car physics, so at least 10 months. I’ve reported it before but this was the first time I created a clear repro case.

Cheers!

[ContextLost]

Looks like the physics engine working as intended…

This is a feature called “physics throttling”. If the physics engine detects that it can’t keep up with realtime it starts stepping the world less frequently and everything appears to move ~4x slower. Some bodies, like Humanoids are never subjected to throttling.

We do this to prevent going into a death spiral, getting further and further behind realtime.

[Jamie_Fristrom]

If I understand you, then, I guess this is more two feature requests than a bug report. Even assuming the jeep scripts are broken and if they were correctly written the jeep would perform fine, just slower, slower isn’t a great result.

So request #1:
Obviously the physics steps have to happen less frequently on slower machines so one feature request I’d like to make is that the various subsystems in Roblox respect deltaT, so parts moving at a given velocity would get from A to B at roughly the same time even on poor machines. Not just velocity but movers and I’m not sure what else would have to be rewritten and I can see how that might be an enormous, risky task.

And request #2:
Fix the sample jeep scripts so that while the jeep might move a lot slower on a slow machine, it at least doesn’t lose control or behave significantly differently than on a fast machine. I’d like to see how the scripts are supposed to be correctly coded - I did find a couple of places where deltaT wasn’t respected there but nothing that explained to me why the thrusters seemed to go crazy when physics was slow.

Frankly, I’m not convinced - I have a feeling that some Roblox subsystems respect deltaT and others don’t (besides character movement). I’ll try to build a repro case for that someday.

[ContextLost]

To do #1 we’d have to have a variable timestep which causes numerical error and instability issues. Slowing down time is unfortunately the best we can do without making things more non-deterministic. You’ll notice the documentation for most physics engines recommends against using a variable timestep…

You’re right that throttling doesn’t affect all systems; there are some bodies that are always realtime. Mostly just things with Humanoids in them. Humanoids are never throttled because it would be a terrible user experience. Unfortunately this also depends on the state the Humanoid is in, and it all makes compensating for this even harder.

There’s Workspace:GetPhysicsThrottling() which returns an integer value between 0-100 which is a percentage of realtime that normal priority bodies are currently being throttled to. The step value give to RunService:Stepped callbacks is only the unthrottled value.

I’m discussing this with the rest of the physics team now. We’re going to need to add some better APIs to make compensating for this easier. Something like a stepped callback that includes both the throttled and unthrottled dt and a function to determine which should be used for a given body.

Looks like the cars in this sample are driven by a BodyPosition setting the target position in front of you every frame. Not sure why this blows up when throttling kicks in, but this is a really silly way to power a car that makes controlling speed difficult. We also need to provide a better racing game sample…

[Jamie_Fristrom]

Hmm, I’ve used variable time step in every engine I’ve worked on or with my entire career (Unity supports either; I switched back and forth with Energy Hook but eventually went with variable) though I’ve never worked with Unreal / Havok, so I don’t know what the preference is there. Fixed timestep strikes me as a great call for single player console games (though we still didn’t do it that way with the Treyarch engine) but I would imagine that with the unpredictability thrown at us by having client-server; handing physics from one machine to another as entities move around; plus some entities on one timestep, other entities on another; and the desire to support an enormous range of platforms, determinism went out the window long ago and I don’t know what the advantage to fixed time step is in that situation, but it still has all the disadvantages, like the game slowing down at ‘low’ framerates (I don’t consider 30 fps slow); one player’s vehicle moving slower than another’s because they’re on a crappy machine; and additional cpu cost if you’re doing more than one physics step per render. Maybe there’s a compelling reason I’m not thinking of though, I often fail to think things through all the way.

Here’s another feature request, if we’re going to keep the physics step separate from the render step: can we get something like game[“Run Service”].OnPhysicsStep - I gather that Heartbeat is not that.

[Khanovich]

I doubt that any game you worked on had variable physics time-step. When you think of a physics Time-step, you think of how much “time” is simulated by every physics frame.

ROBLOX internally runs at 240Hz, and this value is kept separate from the rendering frame rate. There is a caveat that if the game falls below 15FPS, we no longer try to run physics at real time because we make the assumption that if you’re maintaining 15FPS you probably don’t have enough processing power to keep trying to run physics at real-time anyway. ROBLOX running at 30FPS will do 8 internal physics steps every game-frame. ROBLOX running at 15FPS will do 16 internal physics steps every frame.

Running physics at 240Hz means that every physics “frame” simulates about 4ms of time. Think about what this means.

• From a collision detection standpoint this means that any objects that travel so fast that a 4ms window might be enough for them to completely pass through each other, means that fast collisions like this may be completely missed.
• From a Physics Solver (Aka, the thing that at the core takes all the collisions, constraints, and motors and attempts to figure out what is the correct way to physically move things) perspective, you have to correct 4ms of potential errors that have accumulated due to motion to match the conditions set by your collisions, constraints and motors.

Allowing any form of VARIABILITY in this system during runtime creates incredibly unpredictable environments. 240Hz solver already has issues where a 45 degree rotation in a single frame has issues with stability because the object is essentially “oscillating” too fast for the corrective solver to properly maintain. If for some reason we suddenly switched to 120Hz simulation (8ms timestep) this would immediately make the situation WORSE, where only < 22 degree/frame rotations are stable.

I can imagine engines allowing you to chose to run the ENTIRE game’s simulation at lower frequencies, but NEVER vary during run-time.

Engines probably let you run game logic at variable time-step, but NOT physics simulation itself.

Also we do have RunService Stepped: Documentation - Roblox Creator Hub

THROTTLING

That being said, back to the issue of Throttling. Throttling is necessary for the ROBLOX platform because it alleviates some hardware unpredictability for various games. If Throttling didn’t exist, users would start to experience lower frame-rate and unresponsive characters in others. Responsiveness is very important which is why player-character simulation is never throttled.

Now…
As users use more and more physics in their games I’ve seen a massive need for correct interaction with ROBLOX physics, and throttling makes this difficult because its not transparent AND uncontrollable.

I’m currently working on a proposal to do 2 things.

1. Expand RunService::stepped API to return an extra value “throttledStepDt”, which will tell you how much time actually passed on objects that COULD be throttled during that time frame. This way you would be able to know exactly how much time an object simulated between the last RunService::stepped and the current one.
2. Force certain objects into the “Real Time” category that ignores “Throttling”. This would allow racing games to set their cars to “Real-Time” making sure that the cars are simulated at FULL SPEED at all times, while less important things like debris can still slow down.

This doesn’t have an ETA just yet because I’m not exactly sure what form the second API will be in and I have to run by some people to figure out what is the best way to implement it into ROBLOX, but know that I feel like this is a high priority item.

If you have other game examples that are inhibited by throttling PLEASE PLEASE list them here because I want more gameplay examples that get messed up by this!

[Jamie_Fristrom]

With sentences like “I doubt that any game you worked on had variable physics time-step.” and “Think about what that means” and “Responsiveness is very important” you are coming off as pretty darn condescending.

I’m guessing I’m older than you and out of touch with the latest; I’ve certainly never worked on an engine that supported motors or hinges or axles.

Yes, when I said physics I meant what you call game logic. Sounds like by your definition the Treyarch engine didn’t have physics at all, though maybe you’d count the ragdoll (which was variable timestep fwiw but also very sketchy.) And looking again at Unity you’re right about the physics timestep, it just didn’t particularly matter to me because all I was simulating was one character.

Anyways, just want to be heard there - I realize I’m not furthering the content of the conversation any with the above.

Glad to hear about the potential future improvements you’re suggesting.

And thanks for explaining what’s going on under the hood in detail - somebody should probably put that up on the wiki somewhere, along with a clearer explanation of the difference between heartbeat, renderstepped, and stepped - I don’t know where I read it but I thought that heartbeat and stepped had been deprecated and just did the same thing as renderstepped now.

And glad you appreciate the bug report! I was starting to think I shouldn’t have bothered.

Cheers!
Jamie

[Khanovich]

@Jamie_Fristrom

With sentences like “I doubt that any game you worked on had variable physics time-step.” and “Think about what that means” and “Responsiveness is very important” you are coming off as pretty darn condescending.

If that came off as condescending it was not my intent. My post was in the format that I would deliver to any other engineer at ROBLOX. If you look at my post history I try to shine some light on specifics of the engine and physics simulation in game engines in general as much as I can due to the fact that majority of the ROBLOX developers are not 10+ year industry veterans.

The system I am proposing should resolve the issues you brought up. As far as documentation, you raise a good point and we’ve been trying to improve the state of the wiki and have been for the last 4+ years.

There are a lot of reworks going on under the hood, like the one I mentioned, so as we’re replacing more and more systems with more optimal ones I will see if I can push forward a TLDR summary of all the engine-level events and specifics about simulation.

While we’re here, can I get your feedback on something?
Resolving this real-time / throttling issue should give users all the tools they need to make their games properly function in frame-rate independent scenarios. Do you agree or do you still feel like something is missing?

[Jamie_Fristrom]

Thanks!

And thanks for asking. I think something is missing. That thing might just be vehicle sample code that works in a frame-rate independent scenario.

But also I was just experimenting with stepped vs heartbeat vs renderstepped and here’s what I found. I wired up this localscript:

game[“Run Service”].Heartbeat:Connect( function() print(“Heartbeat”) end )
game[“Run Service”].RenderStepped:Connect( function() print(“RenderStepped”) end )
game[“Run Service”].Stepped:Connect( function() print(“Stepped”) end )

And this is what I saw:

You can see that Stepped is not a fixed timestep here, it’s in lockstep with the rendering. So the vehicle script which adjusts the vehicle’s velocity and movers is going to be called at a different rate from the fixed physics steps, and might be part of the reason the vehicle is behaving so differently at low framerates. Unity provides ‘fixedupdate’ and I think Unreal has something similar.

Thanks for asking!

[buildthomas]

Not really related to some of the discussion above, but thought I should mention: RenderStepped, Heartbeat and Stepped will fire every frame. The difference between these methods is their placement in the pipeline. RenderStepped fires right before rendering, and will block the entire pipeline until it is finished executing. Heartbeat and Stepped run in parallel to the frame being rendered, and the difference between those is that Stepped runs right after animations are updated and before the internal physics step, while Heartbeat runs after the physics step.

(this pattern repeats itself for every 2 frames onward)

You see this back in your screenshot. Every frame, RenderStepped fires first, then Stepped, then Heartbeat.

PS: Yes they should say more about the pipeline and where certain signals fire on the wiki. AFAIK the info above is not on the wiki (someone correct me if I’m wrong though, but I couldn’t find it).

[Khanovich]

Yeah as the above user said, technically all of these events are tied to frame rate because of how the engine is structured.

In ROBLOX engine we have a concept of frame which usually entails this:

1. Rendering [Rendering is only #1 because we make sure every frame tries to start every 16.6666ms, and starting with rendering results with smoothest visuals]
2. Incoming Networking Data
3. GameLogic
4. Physics (this has a variable # of “WorldSteps” that maintains itself at 240Hz regardless of Framerate, and does the throttling we discussed above).
5. Heartbeat
6. Outgoing Networking Data

It sounds like what you’re asking for for an event that fires for each of those 240Hz internal physics steps (the “WorldSteps”) as I called them because the only events that are available to users are the ones that can vary in frame-rate.

Is there a reason you can’t run your control code at a variable frequency for car Control? The reason we only expose these “frame” events is because user input is tied to Frame Rate, so the user doesn’t have the capability to modify the events between frames anyway.

Exposing these internal Physics Steps (World Steps that run at 240Hz) would also be hard because they make the assumption that the data-state of the world isn’t modified in between the frames once these World Steps start running.

What are the cases you would want to modify something every one of those 240Hz?
Usually if someone has a control script that modifies Velocity on Body Velocity, the control script should take into account how much simulation time has passed between now and the last time it was run and adjust the velocity according to the time passed.

[Jamie_Fristrom]

I’m thinking more like every n physics steps, probably 4, the fixedupdate event would fire.

One of the things I think is happening with the racing sample jeep is the vertical thrusters (which use a BodyThrust, lines 103-117 of Jeep.CarScript.LocalCarScript) are updating themselves after more physics steps on slow machines, the result being they thrust longer before adjusting, which I believe accounts for why the vehicles are more bouncy at lower framerates. (If we ignore throttling, a machine running at 30-40ms cpu would have 8-12 frames of physics pass before the thrusters get adjusted whereas a machine running at 16ms cpu would only have 4.) It’s not as simple a problem as dv = a*dt and I personally have no idea how to approach it or if it’s even possible to solve. But if we had a fixedupdate I wouldn’t have to solve it and would feel comfortable about using the existing jeep hovering mechanics.
(There are other places in the jeep where dt is ignored, such as line 164, and that is an easy fix, but I don’t know how to solve the thruster problem other than ‘stop using thrusters’)

[Khanovich]

Yeah I see your point. Not even from a perspective of user control, but actually modifying various damping values and force applications requires a higher and more stable frequency. It seems like something like this would benefit from being checked every time an internal physics step fires, rather than only when user can actually modify input.

I’ll start a conversation on this, we’ve talked about something similar before, but exposing internal physics steps to script events would be even hard to implement then the things I mentioned previously because of the expectation that some data does not change between two different RunService steps.

In general our approach has been simply adding more macro-tools/objects in game that should handle anything like this that you need under the hood, but as games get more advanced being able to bind events to individual internal “WorldSteps” may become more necessary.

[Jamie_Fristrom]

Thanks! I feel listened to now. <3

[Tomarty]

@ContextLost

Some bodies, like Humanoids are never subjected to throttling

My game uses custom character physics, and low end mobile devices have a common problem where characters walk and fall extremely slowly (which I assume is causes by throttling.) How do I prevent my custom humanoids from being subjected to throttling?