Compatibility replacing Legacy on 17th of June 2019

[zeuxcg]

So let me start by saying that we actually read things that you post Feedback is welcome, even if it’s negative (but please be polite and respectful of the work we’ve done), but keep in mind that we may disagree with some of it and as a result not act on it.

Why did we introduce Compatibility if it’s so obviously different than Legacy and looks worse? Well, looking better was never the point of Compatbility. Let me walk you through the decisions we’ve made and explain why we made them.

Roblox in the old days had very primitive lighting. I’m sure some of you remember the days where there was just the sun with no shadows - I surely do because my first project was to build a dynamic lighting system. (stencil shadows don’t count, ShadowMode: All was not available in-game as an official feature) The foundation of the lighting system is a rendering equation, and the one we used worked like this: it took the color of the part, took the lighting value computed using directional light, and multiplied the RGB values together. If lighting was 1 (which you could easily get by maxing out Ambient or Brightness), you got the exact same color you used - plain and simple.

Thing is, real world doesn’t work like this. When we built the original Future Is Bright prototype, we very quickly realized that to get where we need to be in the future, which is “awesome visuals”, we need to use a more realistic model of the real world lighting equation. This needs two foundational components:

• High Dynamic Range. This means intermediate values aren’t clamped to 1, which is why you can get high contrast between lit and shadowed areas.
• Linear space lighting. This means all equations must be performed in a different space compared to sRGB that you’re used to seeing in color pickers et al - in this space, values correspond to amount of physically emitted light in a linear fashion. I don’t want to go in too much detail about “why” here, but there’s a lot of good material on the web, in particular Linear-Space Lighting (i.e. Gamma) – Filmic Worlds (take a look at the image in the beginning - this shows that in the sRGB space you’re all used to, the color “in between” 255 and 0 perceptually is not 128 - it’s 192! 192 maps to 0.5 in linear space, making the math correct.

Once you have to have high dynamic range, you also have to have a tone mapper - it remaps higher-than-1 values to 0…1 range that can be displayed on conventional display panels. The process of making a tonemapper is on the intersection between art & science, but we had to make one - it’s a critical piece of the system.

And this is almost all there is to it about Voxel. We need HDR. We need linear-space lighting. We need a tone-mapper. Once you accept the fact that you need to model real-world optics, you just can’t get away from these things. The only other component of Voxel is anisotropic occupancy - it’s an extension to our voxel grid that makes it so that thin parts don’t let light through. (the current implementation is limited in that it still lets light leak through corners which is something we want to address in the future)

This is also where concerns about “vibrant colors” come in. Take a look at this picture from Unity’s docs:

Do colors in gamma space look more vibrant at Intensity 1? Uh, I guess. But that’s not how the real world works. And higher intensity values result in really blown out images, which is bad when you start talking about HDR and modeling real-world lighting values.

This is why we made Voxel. ShadowMap is based on the exact same math. This is the foundation for our future improvements that add light shadows and more precise local lights as well (aka FIB v17).

We strongly believe this is how content should be built. We strongly believe content looks better within this new mathematical formulation. It seems like the majority of the community agrees - you can get much better quality with the new system.

Voxel isn’t perfect. There’s some technical limitations wrt shadow fidelity, light leaking; there’s also issues with specular highlights that fundamentally stem from the fact that when you’re modeling real world, but every single surface with a custom texture behaves like painted plastic, things don’t look so good. We will be addressing these with future updates but even given these limitations, Voxel is a superior system.

Which brings us to the question of existing content. (and I need a break)

[zeuxcg]

Ok, so the real difference in Voxel is that it’s a different mathematical foundation. Rendering pipelines are like 50% math; by changing the fundamental assumptions about the color space you work in, you have to redo a lot of the work. We had to very substantially change the water shading for example because the way things were done before were effectively hacked together up until they look right; it’s harder to do this when the underlying foundation is sane, you have to be more rigorous about your derivations.

This mathematical foundation is crucial for all followup work that we do - so all future development will happen in this framework. However, when you change the mathematical model, but keep the numbers that are being fed into it (which in our case is the content that you have made - part colors, lighting settings, textures) - you get a different result. Since content is often tuned to the current behavior without regards to whether the values are physically correct, naturally just flipping the switch doesn’t work. This is why you need to tweak a lot of settings to make your levels look reasonable when switching to Voxel.

We need to have the existing content work. How do we do that? The obvious solution is to keep Legacy forever. There’s two big issues with that.

One, it’s a constant tax on maintenance on existing system and development of new features. This isn’t exclusive to this project - I mean, you’ve heard it all before, this is why we removed old terrain, this is why we removed old physics solver. For example, we had to redo the water shader for the new model - the old one is still around to support legacy. We had to redo postprocessing pipeline - the old one is still around to support legacy. Whenever we implement new materials - forcefield, for example - the development time increases because both need to work. Whenever we change existing features, both need to work. But they are so different that it takes time to do that.

Two, it’s a tax on performance as well. Because of teleports and some other constraints, we need to be able to switch between these often; there’s some restrictions that this imposes on some systems that I won’t go into - suffice it to say that the option that we had to choose to not go completely crazy wrt maintenance is to emulate parts of the old pipeline by doing extra work in our shaders.

When we originally released Phase 1, we actually didn’t duplicate some things like the post-processing pipeline - large parts of it were shared between the two systems, and could be configured to work in either mode. This code runs on your GPU and has to process millions of pixels/vertices - the price for even small things is very high. This is part of why phase 1 took more time to ship on all platforms than we thought - we had to go back and completely fork the post-processing pipeline to mitigate some of the performance loss. This is because, when you emulate both systems on the GPU, your cost isn’t equal to the cost of the currently selected system, and it’s not even max(system1, system2), it’s somewhere between max(system1, system2) * overhead and system1+system2 depending on the case.

After we fixed that by duplicating the entire postfx stack, we still emulated both systems within the main rendering passes - these are much harder wrt maintenance for us to duplicate. This is where the resulting ~5% performance overhead comes from (it’s actually more significant on lower-end devices where it hurts the most).

So it’s a big issue that we have both systems - not because we have two of them, but because they’re so different.

A good counter-example is Voxel vs ShadowMap. You’d think that it’s hard to maintain both, but we need Voxel on lower-end platforms that don’t support ShadowMap and low quality levels for players with weak hardware, so we don’t lose much by supporting Voxel as a dedicated mode. The only issue is that we ended up with two shadow systems (Humanoid shadows in Voxel mode and full scene shadows in ShadowMap mode), but I’m hoping we will find a good way to merge them, in effect using ShadowMap only on your player character in Voxel mode in the future.

Keeping both systems is problematic. Historically the way we’ve dealt with this is wait for developers to migrate (thank you to so many of you who have migrated their games to Voxel!), announce the complete switchover some time in advance, switch and assume that most content works okay and the content that doesn’t can be migrated at some point later. This wasn’t appropriate in this case.

edit: I’m blocked by 3 consecutive replies rule, so I’m waiting for somebody else to reply to this thread to post the final part of this really long update.

[gluGPU (Glu)]

Great update, Roblox! I look forward to what people will create with this new lighting system!!

There you go, @zeuxcg.

[zeuxcg]

The reason why system migration takes a while in Roblox is because usually, when you switch from one system to another, you’re going to have behavior differences; no matter how good or bad they are, they can break games. The games become unplayable. This is a problem. This usually doesn’t happen for rendering features - one day outlines will disappear from Roblox (sorry) but it’s hard to imagine many games that would become unplayable as a result. Lighting is different.

There’s three common issues we’ve seen when just switching to Voxel:

1. Areas that used many lights with extremely high brightness were just white. Old system didn’t support HDR so it clamped the impact of all lights to 1. New system supports HDR so it faithfully reproduces the incredibly bright color which then gets transfered to the screen, but the tonemapper can’t figure out how to expose this (have you ever tried to take a photo of a large dark building against bright sky? You’ll know what I’m talking about - sky will turn white because the image is overexposed for the sky, or building will turn black because the image is underexposed for the building). This breaks gameplay! You can’t see things anymore.

2. Areas that used buildings with thin walls without any lights inside were just black if Ambient was 0. Old system didn’t support thin walls. If the walls were thicker, the place creator would put lights inside the building, but the walls were thin so they didn’t feel they have to. This breaks gameplay! You can’t see things anymore.

3. Games were generally darker than expected. This is because of the math difference between linear space lighting and sRGB lighting. This is why we recommend increasing Brightness a bit during migration. This rarely breaks gameplay, but is inconvenient.

Issues 1 and 2 were relatively common. This means we couldn’t treat this as a rendering feature for the purpose of migration since it effectively breaks behavior by rendering (see what I did there?) the game unplayable.

However, the performance and mainteinance overhead is really important. We want to release more updates to rendering that have a bit more rendering cost, and yet we can’t fix the 5+% cost of having to keep Legacy afloat. We want to release more rendering features and yet for many we need to do extra work to support Legacy. (in fact, easily half of the bugs in the initial ShadowMap release happened because we still have Legacy!)

This is what Compatibility solves. It fully solves problem 1 and 2, and mostly solves problem 3. Problems 1 and 2 are solved by clamping the output of the lighting system to 1 (which is almost free) and disabling anisotropic occupancy (which is free); problem 3 is solved by adjusting the input parameters - light color, intensity, bloom strength, tonemap parameters - to try to match Legacy better (which is free).

Here’s what we get as a result.

Existing places don’t break when transitioning from Legacy to Compatibility. They don’t look the same - mathematical models are still different! - and they don’t look better - we can’t retune your content for you! - but we make sure that if you could play the game before, you still can, by eliminating the gameplay-breaking parts of the update.

We don’t have to pay the performance penalty. All existing places get a slight boost in performance for low-end systems.

We almost don’t have to pay the maintenance penalty. There’s a very thin layer that adjusts input parameters, and that’s basically it. Not quite zero cost but really close.

The expectation is that developers that want their games to look great will learn to work with the new system, and if they need to exercise artistic control to push their game to have more vibrant colors etc., the hope is that between lighting controls and color correction effect there’s a fair amount of options to play with to reach good results.

The expectation is that there will be developers who don’t have the time or desire to update their games, and these games will look a bit differently - maybe worse than they looked before - but will still be perfectly playable, so players will still enjoy them. This is the compromise - we can’t guarantee the same look, but we can try to make sure you don’t lose players.

As you can probably tell, Future Is Bright is a really ambitious project. The way to ship ambitious projects is to do it in small chunks (which is why we’re splitting it into so many phases) and to try to execute them quickly - as much as we’re able to. We really want to give you even more cool features to play with but the performance issues (that we have to be very serious about when making these changes) and the maintenance issues hamper our progress. This was a big concern with Legacy - we could never replace Legacy with Voxel due to aforementioned gameplay breaking issues. But with Compatibility solution, it seems like the real issue is that some developers are attached to the Legacy look and aren’t able to reach the exact same visuals with Voxel.

We can’t wait forever for Voxel and future technologies to become “perfect”. And it’s not clear that we can make everybody happy no matter how many updates to Voxel we do. Which is why we decided that the state of Compatibility is good enough for players to still enjoy your games, and to sunset Legacy to make sure that its presence doesn’t impact current and future work.

Thank you for reading.

[Swinsor (Ashley)]

@zeuxcg thank you for clearing up any confusion! This was very helpful! Thanks!

[x86_architecture (TritiumAndDeuterium)]

Is Shadowmap still only available to studio? And if I publish something that is in shadowmap will the game be automatically be converted to voxel?

[Muoshoob (Officer_Bacon)]

If nothing goes wrong, ShadowMap should be rolling out to the client today or tomorrow. (pls correct me if im wrong @zeuxcg).

[Mystyle48]

Shadowmap is still only a studio thing, and your game will be converted automatically to compatibility if published with Shadowmap enabled.

[ConvexRumbler]

Update: Forced compatibility is postponed due to the bug in UI system. Will post new date here soon.

[bachovar123 (Bondoman)]

Good to know.
i personally allwys preferred legacy lighting over everything else

[anon82070253]

Shadow Mapping is far superior to legacy.

[28Pixels]

Could you explain why you prefer legacy over the other Lighting.Technology options?

[Maximum_ADHD]

I mentioned this in another thread, but I’ll post it here too.

For anyone who wants Roblox’s legacy tone-mapping while using shadow maps, I found a ColorCorrectionEffect configuration that gets it pretty much on-point.

[GeorgeOfAIITrades]

For day time it’s spot on, but night time it becomes awfully hazy.

It’s not quite legacy, but removing some of the negative Contrast is a big help.

[Maximum_ADHD]

I think this can be fixed by adjusting the contrast based on the Y-value of the sun’s direction. I’ll look into it more when I get the chance.

[JuuzouBlitz]

“not nicely compatible with the new lighting system” What does that even mean? legacy is a different lighting system than voxel, why does it need to be compatible. It worked just fine before they introduced voxel or anything else.

[PeZsmistic]

These quotes are relevant.

@ConvexRumbler

Having legacy in our code base has couple negative points

• We have to dedicate our engineering time to maintain the old code. We can use this time better, like working new features for You
• Having Legacy and Voxel is causing ~5% rendering slowdown across all platforms. Lets get those 5% percent back

@zeuxcg

We need to remove Legacy. It’s problematic because it requires us to maintain a lot of configurations that are fundamentally different from what we want to have; the need to be able to switch technology between games during teleports requires us to inject some code in the shaders that can seamlessly switch between “old” and “new” math even on lowest quality, and we pay a performance penalty for this.

@zeuxcg

we won’t spend much time testing legacy mode and might need to start introducing small breaking changes to optimize Voxel pipeline in places where two pipelines share code. This won’t happen immediately upon release of course, and we plan to keep Legacy around for some time, but at some point Legacy levels will start changing behavior.

These three posts by Zeuxcg are interesting if you’re curious why.

Compatibility replacing Legacy on 17th of June 2019

So let me start by saying that we actually read things that you post Feedback is welcome, even if it’s negative (but please be polite and respectful of the work we’ve done), but keep in mind that we may disagree with some of it and as a result not act on it. Why did we introduce Compatibility if it’s so obviously different than Legacy and looks worse? Well, looking better was never the point of Compatbility. Let me walk you through the decisions we’ve made and explain why we made them. …

Compatibility replacing Legacy on 17th of June 2019

Ok, so the real difference in Voxel is that it’s a different mathematical foundation. Rendering pipelines are like 50% math; by changing the fundamental assumptions about the color space you work in, you have to redo a lot of the work. We had to very substantially change the water shading for example because the way things were done before were effectively hacked together up until they look right; it’s harder to do this when the underlying foundation is sane, you have to be more rigorous about y…

Compatibility replacing Legacy on 17th of June 2019

The reason why system migration takes a while in Roblox is because usually, when you switch from one system to another, you’re going to have behavior differences; no matter how good or bad they are, they can break games. The games become unplayable. This is a problem. This usually doesn’t happen for rendering features - one day outlines will disappear from Roblox (sorry) but it’s hard to imagine many games that would become unplayable as a result. Lighting is different. There’s three common iss…

[AureateGold]

Well this is a perfectly prime example of rushing into things without any idea of what could happen.

(Anyone remember the compatibility mode fiasco)

Either way, this is mainly my opinion:

Yes, the new FIS (Future is Bright) system is looking pretty dandy from messing around with it, but like all of the other features each one is special and to be used upto the developers idea, and should they want legacy in my opinion they should be able to have it.

Getting rid of legacy lighting is like getting rid of a failed social media post, your trying to cover it up and that’s what in my opinion I believe Roblox are doing.

My personal opinion: Keep legacy and just keep working on shadowmap and let the developer have choice on what lighting system they wish to use.

[Rocky28447 (Rocky)]

Forgive me if I’m wrong, but your post does not even seem to consider the negative performance impact (which, iirc is quite considerable) of Roblox supporting 2 separate lighting engines.

Compatibility exists for a reason. It exists for those who want the look and feel of legacy (no HDR, no shadowmaps, etc) albeit with slightly different colors. I think the people who still want legacy can learn to live with slightly different looking colors so nobody has to deal with the negative performance impact.

[unmiss (Jack)]

Did you even read any of the posts quoted in the post right above yours? It explains everything. Frankly your comment is very shallow to the reasons needed for removing Legacy