Stencil Shadows Implementation

[Razorboots]

Roblox 2016-2022 Multi-Surface Shadow Engine

Notice (For those with performance issues & complaints)

I’ve gotten a lot of feedback on the state of this system and I feel I need to clarify some limitations. This system was created in 2014 and 2016 Studio for a project my friend was working on, though this version is updated to be functional with modern Roblox.

For future reference, I am currently creating a game using OGRE3D and doing my best to understand their stencil shadows implementation to hopefully create a more accurate and performant version of this in the near future!

This engine will likely be laggy on your system with a large number of parts. In other game engines, shadow volumes/stencil shadows is a system that involves mapping vertices of every mesh in your scene to determine which pixels on your screen are occluded, and which aren’t. This process is made blazing fast due to shaders, which are programs that run on the GPU and are executed in parallel on the GPU. Roblox’s previous rendering engine, based off of OGRE3D, had a native implementation of shadow volumes for directional, spot, and point lights. I’m guessing sometime around mid-2014 when Roblox rewrote their rendering engine, this feature was removed and voxel lighting became the only shadow implementation other than iirc, shadow maps, which were used on models with Humanoids until shadow maps became the default for all entities with FIB 3.

Unfortunately, Roblox doesn’t have shaders implemented into Studio, so shadow volumes are not possible to do on Roblox by traditional means.

I’ll address a few complaints I have received regarding this implementation:

• Why do the shadows have visible polygon edges?
  – This is a result of SurfaceGui size limits and mipmapping. You may notice that as you get close up to a shadow, the creases are rarely visible, but the further you go out the more visible they become. Mipmapping is the result of the rendering engine automatically trying to smooth out textures to prevent aliasing.
  – The overlap amount can be edited in the Triangle module under the Shadow module using the variable called extra.
  – You can also change the color of the shadow to a solid color and make the transparency black, you can find these options as child objects of the Shadow module.
  – In a future update, I may potentially implement a system that calculates the color of the part a shadow canvas is attached to and uses that color while making the shadow opaque.

• Why do meshes and wedges have boxy shadows?
  – Unfortunately, meshes cannot accurately cast shadows because Roblox doesn’t have a method to extract the vertices from a mesh, making them not viable. I suggest outlining meshes with invisible parts to achieve shadows for them.
  – As for “Sphere” SpecialMeshes, I can potentially write up a system that calculates vertices based on the size of the parent Part in a future update.

• Why is it so laggy?
  – Unfortunately lag is unavoidable in a scene with a lot of parts, and there are a few reasons for this.
  – This implementation performs several calculations that fires a ray through every edge-detected vertex of a part, projects the resulting point to the surface of every shadow canvas that the shadow should be visible on, transforms the resulting projected points to 2D triangles, and clips these triangles by the size of the shadow canvas.
  – This process is all executed on the CPU and there is unfortunately no method to utilize the GPU’s parallel computing power to compute these calculations.
  – Fortunately, Roblox does have CPU parallel computing (Parallel LuaU), but this won’t entirely fix the lag. Similar to my baked lighting implementation, In a future update, I plan to implement parallel LuaU to hopefully smooth out the lag and allow the ability for more parts to be used.

If you have any other questions, feedback, or suggestions, feel free to let me know. Any feedback contributes toward making this implementation more viable for developers!

Description:

Roblox used to have an old shadow rendering method called Stencil Shadows or Shadow Volumes before they created their own rendering engine, which was called Ogre3D. Ogre3D has an implementation for Shadow Volumes, but soon the feature was removed in place of Voxel shadows in Roblox.

This engine is an attempt at recreating Stencil Shadows in modern Roblox that are complex and include multi-surface shadows, which is often an implementation that’s lacking in many raytracing-based shadow algorithms.

The source for this project is here: GitHub - Razorboot/Roblox-2016-Shadow-Engine

Built off of Egomooses Shadow Silhouttes Tutorial: Silhouettes and Shadows - Scripting Helpers

• This is an attempt at an optimized raycasted shadow-polygon implementation that’s designed to look like Roblox’s old Stencil Shadows system!
• This is a complex set of object oriented mini-classes and polygonal algorithms for:
  • accurate n-gon to triangle conversion,
  • clipped polygons,
  • multiple light sources,
  • optimized vertex grabbing,
  • ray to plane intersection,
  • rotated shadow occluders and canvases,
  • world position shadows to 2d space, clipped, triangle-based shadows,
  • surface-based real-time lighting calculations:
    an attempt to make the system appear more well-integrated with shadow engine.
• Inspired by early 2000’s Shadow Volumes in games like Thief: Deadly Shadows and Doom II, as well as Ogre3D (Roblox’s old rendering engine).
• Manifold data-structure system was inspired by collision manifolds in AABB physics.
  This implementation is classless and includes nested arrays as a form of orginization.

New Changes as of 12/14/22:

• Complex multi-surface shadows supported.
• Rotated parts (occluders and canvases) now supported.
• A mini lighting engine for rendering part surfaces is now added and functional in real-time!
  • This supports multiple lighting options you now have control over in Shadow script!
• Shadows are now Surface-Gui based for more optimal results.
• Global and Local lighting now supported.

Diagram: A simplified view of a plane and an occluder:

• 

Demos: Some test-scenes I made to showcase a few features the engine has

• 
• 
  

Getting Started:

If you plan on using the Engine itself:

1. First, download the Shadow_Modules.rbxmx file.
2. If you plan on using the engine client-sided, right click on StarterGui, select Insert From File, and select the Shadow_Modules.rbxmx file.

• Keep in mind FilteringEnabled will have to be enabled in order to use the client-sided engine.

3. If you plan on using the engine server-sided, right click on Workspace, select Insert From File, and select the Shadow_Modules.rbxmx file.

If you plan on using the place file:

1. First, download the ShadowTest_GloballLight_Demo.rbxl file.
2. If you plan on using the engine client-sided, go into ServerScriptService → Scripts → Main.
3. Cut the code from this script into a LocalScript and read the comments inside of the script to know what to uncomment.
4. Place the new script back into the Scripts folder.
5. Next, you can drag both Modules and Scripts into StarterGui.
6. By default, the rbxl file is already set up to be used server-sided.

Include the Shadow module:

1. For client-sided, insert a LocalScript in StarterGui.
2. For server-sided, insert a new Script in Workspace.
3. To finally include the module, type:

--# Include
local Modules = --Location of the inserted Shadow_Modules.rbxmx file
local Shadow = require(Modules:WaitForChild("Shadow"))

Creating Light and Shadow Instances:

Shadow Canvases:

• Shadow Canvases are SurfaceGui’s that are mapped onto parts!
• They are the surfaces of a part that can have shadows projected onto them.
• Shadow Canvases are flat planes that can be applied to any face of a Part.
• Shadow Canvases can be added to a part or model using:

-- Add canvas to all surfaces of a part
Shadow.setModelProperty(Part, "isShadowCanvasAll", true)
-- Add canvas to all surfaces of a model
Shadow.setPartProperty(Part, "isShadowCanvasAll", true)

• Setting the final parameter to false will delete a pre-existing canvas.
• "isShadowCanvasAll" is one out of 6 surfaces. You can replace All in isShadowCanvasAll with either Top, Bottom, Right, Left, Front, or Back in order to add a Shadow Canvas to a single surface.

Occluders:

• Occluders are the Parts that block light sources, and thus cast shadows onto shadow canvases.
• All occluders are treated as cubes, so any other shaped-part will be treated as such.
• I’m currently working on adding more Part-types, so this won’t be permanent!
• A part or model can be set to an Occluder using:

-- Set a part to an Occluder
Shadow.setModelProperty(Part, "isShadowOccluder", true)
-- Set a model to an Occluder
Shadow.setPartProperty(Part, "isShadowOccluder", true)

• Exactly as the previous function, setting the final parameter to false will ensure the part won’t cast shadows.

Lit Surfaces:

• This is an optional category that isn’t essential for shadow creation.
• A Lit Surfaces Part allows the brightness of the surfaces of a part to be rendered more accurately to light sources.
• This is my attempt at making part surfaces look less jarring when compared to dark shadows.
• The settings that control the color of part surfaces can be found inside of the Shadow module inside of the inserted Shadow_Modules.rbxmx file.
• You can make a part or model have Lit Surfaces using:

-- Allow the surfaces of a part to be lit
Shadow.setPartProperty(Part, "hasLitSurfaces", true)
-- Allow the surfaces of a model to be lit
Shadow.setModelProperty(Part, "hasLitSurfaces", true)

• Lit Surfaces are stored into containers called LitPartManifolds, this allows the script to easily work with them:

local litPartManifolds = Shadow.getLitPartManifolds()

• Lit Surfaces need to be updated each frame in order to account for changes in position relevant to Light Sources:

litPartManifolds = Shadow.updateLitPartManifolds(litPartManifolds)

Light Sources:

• Light Sources is a list of Lights in your scene that you want to cast shadows and influence Lit Surfaces.
• You can insert all light sources in a container using:

Shadow.getAllLightSources(workspace)

• You can insert and remove a specific light source into your scene using:

Shadow.insertLightSource(light)
Shadow.removeLightSource(light)

• Light Sources need to be updated to account for changes in position and range, you can also update all light sources using:

Shadow.updateAllLightSources()

Root/Canvas Manifolds:

• When creating shadows, each Shadow Canvas is passed into a function that generates arrays with relevant information for the Occluders and Light Sources that will be taken into account.
• To create a root manifold for all Shadow Canvases, use:

local rootManifolds = Shadow.getRootManifolds()

• Root manifolds need to be updated each frame in order to account for changes in Position of the Shadow Canvas itself, Occluders, and Light Sources:

rootManifolds = Shadow.updateRootManifolds(rootManifolds)

Experimental Mode:

• Inside of the Shadow module, you’ll find an option called useExperimental.
• This simply adds the ability for shadows to be rendered when Light Source to vertex rays don’t intersect with the Shadow Canvas plane.
• This option is buggy, so I suggest use it with caution.

[LymthomHD (MrZombie)]

This looks cool but is there any reason to use this instead of Voxel?

[pankii_kust (sillycake)]

Voxel shadow is casted in 4x4x4 grid and ignores wall as it’s made to calculate in grids without raycasting.