Creating an outline/edge shader

I’m in the process of trying to recreate the “outlining” effect many games have on the edges of surfaces. If you’ve ever played Antichamber or similar-looking games to that, that is exactly the kind of effect I want:

images

I also know of someone before me who tried to create this same effect, but although they outlined the process/rules for outlining on that post (which is the exact same as what I want), they didn’t seem to ever figure out how to do it (unless they just decided not to update the post that they had)

Other Guy Attempt

My game is made entirely of regular block parts, with no wedges or anything, which should make it somewhat easier. I already created a way to figure out which surfaces of a part are parallel, but I have no idea how to actually outline parts. Currently, the game is static as well, but eventually, I might add dynamic objects, and I would want this effect to be updatable, so if I could get this outlining to be relatively quick, that would be cool too.

My current code, I should be able to build off of it to get this outlining effect hopefully.

local Vertices = {
	{1, 1, -1},  --v1 - top front right (v2, v4, v5)
	{1, -1, -1}, --v2 - bottom front right (v6)
	{-1, -1, -1},--v3 - bottom front left (v2, v4, v7)
	{-1, 1, -1}, --v4 - top front left (v8)

	{1, 1, 1},  --v5 - top back right (v6, v8)
	{1, -1, 1}, --v6 - bottom back right
	{-1, -1, 1},--v7 - bottom back left (v6, v8)
	{-1, 1, 1}  --v8 - top back left
}

local EdgeConnections = {
	{1, 2},
	{1, 4},
	{1, 5},
	{2, 6},
	{3, 2},
	{3, 4},
	{3, 7},
	{4, 8},
	{5, 6},
	{5, 8},
	{7, 6},
	{7, 8},
}


local faces = {
	Enum.NormalId.Front,
	Enum.NormalId.Back,
	Enum.NormalId.Left,
	Enum.NormalId.Right,
	Enum.NormalId.Top,
	Enum.NormalId.Bottom,
}

-- Get normal of a face on a part
local function GetNormal(part, normalId)
	if normalId == Enum.NormalId.Top then
		return part.CFrame.UpVector
	elseif normalId == Enum.NormalId.Bottom then
		return -part.CFrame.UpVector
	elseif normalId == Enum.NormalId.Right then
		return part.CFrame.RightVector
	elseif normalId == Enum.NormalId.Left then
		return -part.CFrame.RightVector
	elseif normalId == Enum.NormalId.Front then
		return part.CFrame.LookVector
	elseif normalId == Enum.NormalId.Back then
		return -part.CFrame.LookVector
	end
	--return part.CFrame * Vector3.FromNormalId(normalId) - part.Position
end

--Get corners of a part using premade verticies table
local function GetCorners(Part)
	local corners = {}
	for _, Vector in pairs(Vertices) do
		table.insert(corners, (Part.CFrame * CFrame.new(Part.Size.X/2 * Vector[1], Part.Size.Y/2 * Vector[2], Part.Size.Z/2 * Vector[3])).Position)
	end
	return corners
end

-- Gets all corner positions of a part, then uses a premade table to get the corresponding positions of edges on that part
local function GetEdges(Part) 
	local corners = GetCorners(Part)
	local edges = {}
	for _, vertexPair in ipairs(EdgeConnections) do
		table.insert(edges, {corners[vertexPair[1]], corners[vertexPair[2]]})
	end
	return edges
end

--return the correct component of the position of a face on a part depending on what face it is. (also return the full position for debugging if needed)
local function GetFacePos(part, normal)
	if normal == Vector3.FromNormalId(Enum.NormalId.Top) or normal == Vector3.FromNormalId(Enum.NormalId.Bottom) then
		local pos = part.CFrame * (normal*part.Size.Y/2)
		return pos.Y, pos
	elseif normal == Vector3.FromNormalId(Enum.NormalId.Left) or normal == Vector3.FromNormalId(Enum.NormalId.Right) then
		local pos = part.CFrame * (normal*part.Size.X/2) 
		return pos.X, pos
	elseif normal == Vector3.FromNormalId(Enum.NormalId.Front) or normal == Vector3.FromNormalId(Enum.NormalId.Back) then
		local pos = part.CFrame * (normal*part.Size.Z/2) 
		return pos.Z, pos
	end
end	

local function compareFacePos(part1Pos, part2Pos)
	--if normalId == Enum.NormalId.Top or normalId == Enum.NormalId.Bottom then
	--	if part1Pos.Y == part2Pos.Y then
	--		return true
	--	end
	--elseif normalId == Enum.NormalId.Left or normalId == Enum.NormalId.Right then
	--	if part1Pos.Z == part2Pos.Z then
	--		return true
	--	end
	--elseif normalId == Enum.NormalId.Front or normalId == Enum.NormalId.Back then
	--	if part1Pos.X == part2Pos.X then
	--		return true
	--	end
	--end
	--return false
	
	if part1Pos == part2Pos then
		return true
	end
	return false
end

--Get all parts in a table
local parts = {}

for _, obj in pairs(script.Parent:GetChildren()) do
	if obj:IsA("BasePart") then
		table.insert(parts, obj)
	end
end

-- Store faces of each part, the normals corresponding to each face (in case part is rotated), and the edges of each part
local surfaceDescriptions = {}

for _, part in ipairs(parts) do
	local normalsOfPart = {}
	local facesOfPart = {}
	for _, face in ipairs(faces) do	
		local normal = GetNormal(part, face)
		table.insert(normalsOfPart, normal)
		table.insert(facesOfPart, face)
	end
	local edgesOfPart = GetEdges(part)
	surfaceDescriptions[part] = {normalsOfPart, facesOfPart, edgesOfPart}
end

for _, part1 in ipairs(parts) do
	for _, part2 in ipairs(parts) do
		if part1 ~= part2 then
			for i = 1, 6 do
				--Get the actual positions of a face on the correct axis (eg y component if we are looking at the face on the top/bottom)
				local part1Pos = GetFacePos(part1, surfaceDescriptions[part1][1][i])
				local part2Pos = GetFacePos(part2, surfaceDescriptions[part2][1][i])	
				
				if compareFacePos(part1Pos, part2Pos) then --Surfaces are parallel			
					local surfaceGUI1 = Instance.new("SurfaceGui")
					surfaceGUI1.Parent = part1
					surfaceGUI1.Face = surfaceDescriptions[part1][2][i]

					local surfaceGUI2 = Instance.new("SurfaceGui")
					surfaceGUI2.Parent = part2
					surfaceGUI2.Face = surfaceDescriptions[part2][2][i]

					local randomColor2 = BrickColor.random()

					local frame1 = Instance.new("Frame")
					frame1.Parent = surfaceGUI1
					frame1.Size = UDim2.fromScale(1,1)
					frame1.BackgroundColor = randomColor2

					local frame2 = Instance.new("Frame")
					frame2.Parent = surfaceGUI2
					frame2.Size = UDim2.fromScale(1,1)
					frame2.BackgroundColor = randomColor2
				end
			end
			--local randomColor1 = BrickColor.random()
			--for _, face in ipairs(faces) do	
			--	local part1Normal = GetNormal(part1, face)
			--	local part2Normal = GetNormal(part2, face)
			--	local _, part1Pos = GetFacePos(part1, part1Normal)
			--	local _, part2Pos = GetFacePos(part2, part2Normal)	
				
			--	local part = Instance.new("Part", workspace)
			--	part.Anchored = true
			--	part.Size = Vector3.new(1.1,1.1,1.1)
			--	part.BrickColor = randomColor1
			--	part.Material = Enum.Material.Neon
			--	part.Position = part1Pos
				
			--	if compareFacePos(part1Pos, part2Pos, face) then
			--		local surfaceGUI1 = Instance.new("SurfaceGui")
			--		surfaceGUI1.Parent = part1
			--		surfaceGUI1.Face = face
					
			--		local surfaceGUI2 = Instance.new("SurfaceGui")
			--		surfaceGUI2.Parent = part2
			--		surfaceGUI2.Face = face
					
			--		local randomColor2 = BrickColor.random()
					
			--		local frame1 = Instance.new("Frame")
			--		frame1.Parent = surfaceGUI1
			--		frame1.Size = UDim2.fromScale(1,1)
			--		frame1.BackgroundColor = randomColor2
					
			--		local frame2 = Instance.new("Frame")
			--		frame2.Parent = surfaceGUI2
			--		frame2.Size = UDim2.fromScale(1,1)
			--		frame2.BackgroundColor = randomColor2
			--	end
			--end
		end
	end
end

Small Edit: I also was looking into possibly making custom SSAO, and modifying the effect to produce somewhat of an outline? but that probably would be too hard and not worth it since it may not look like an actual outline. Also feel free to suggest completely alternative methods to what I’m doing, I don’t mind rewriting the code or making something new entirely.

1 Like

You can just use the highlight item recently introduced by roblox

The image given doesn’t work with how Highlight currently functions, unless ROBLOX adds a boolean property for highlighting all edges, and not the edges of the mesh from the current camera cframe, this wouldn’t quite work.

Well hello fellow edge detection lover. You were correct, I’ve not updated mainly because I’m still figuring it out but I have learned a lot since I started (like 2 years ago T_T). Here’s a few warnings on your journey.

Depending on how many parts you have, if your outlining each and every part with more parts, a few 100 can easily increase to about 6x that (depending on the scene) which in my experience can be a little hard on the computer. Try to do everything on the server if you can but note the pathfinder service won’t like you.

If your parts have no rotation then this is relatively easy. I don’t have code but it’d be a lot easier then outlining every shape. If your including rotation then it just got a whole lot harder.

Lastly, their is no way to get the desired effect through cheep methods and using parts. You have to find the Union of all the parts manually to get the desired look; though their is an alternative, using meshes, but that doesn’t allow for dynamic changes to the environment.

If you like I don’t mind sharing the code I got now, I would desperately love help with it.

btw, interesting idea to use frames instead of parts, it’d be more complex but probably more efficient.

1 Like

actually, if all your parts are voxels and in grid (like antechamber) you could just check which spaces are filled around the part and calculate what frames needs to be on what faces.

Nice seeing you here! Unfortunate to hear that you haven’t figured it out yet. I’m not using “frames” though, not sure what you meant by that.

The point about having 6 times more parts is a bit unsettling lol, I’m really not sure what you could do about that. Maybe if you could only render parts that are in your line of sight?

Is there really no way to get this effect through just having the parts as is? The code I posted can, given a folder with parts and the script in that folder, put surface guis on surfaces that are flush with each other, which is cool, but that doesn’t actually solve the detecting of edges. The edge case you showed with the black parts in your post with a “t-shape” seems hard to account for as well.

Wish roblox just provided full shader support :upside_down_face:

Like if all your parts are cubes you could put a surface gui on each face to make it look like it has an outline.

Had the same idea, their is a way to find the angle of the camera and only render the parts in the frame of the camera but idk if that’s automatically done, we also couldn’t really know if it works till we get the outline to work so I’ve kind of put it off to the side for now. Note ray casting to each part from the camera to see if the player can directly see it is an absolute no-no since you’d be doing this twice for every part (for each end) and every frame.

You can use legacy outline but it won’t give you the desired effect. I haven’t found an easy way.

If only :sob:

Ohh, that might actually work. I’d just need to know what edge of the part is flush with another part (although at that point, at the expense of performance I could possibly just try to use parts)

rn I’m making a script that would go through everything and create outlines using parts or something like that. If it’s laggy, its laggy, and at that point ill either give up or try to optimize it LOL

1 Like

Goodluck; hopefully you get it. I’m willing to share the code and articles I’ve got if you’d ever want to see them.

Thanks, if I figure it out or need help I’ll definitely let you know!

1 Like

I believe that is the method for legacy outlines, so you can see the code from that, which will probably be very optimized.

Would be nice if I could find said code :smiling_face_with_tear:

I might be able to find it in the coreScripts, I’ll check it out when I can!

–[[
local function GetEdges(part)
local edges = {}
local corners = GetCorners(part)
for _, Vector in pairs(Vertices) do
table.insert(corners, part.CFrame * CFrame.new(part.Size.X/2 * Vector[1], part.Size.Y/2 * Vector[2], part.Size.Z/2 * Vector[3]).Position)
end
for _, edge in pairs(EdgeConnections) do
table.insert(edges, {corners[edge[1]], corners[edge[2]]})
end
return edges
end

local parts = {}

for _, obj in pairs(script.Parent:GetChildren()) do
if obj:IsA(“BasePart”) then
table.insert(parts, obj)
end
end

local function GetNormal(part, normalId)
if normalId == Enum.NormalId.Top then
return part.CFrame.UpVector
elseif normalId == Enum.NormalId.Bottom then
return -part.CFrame.UpVector
elseif normalId == Enum.NormalId.Right then
return part.CFrame.RightVector
elseif normalId == Enum.NormalId.Left then
return -part.CFrame.RightVector
elseif normalId == Enum.NormalId.Front then
return part.CFrame.LookVector
elseif normalId == Enum.NormalId.Back then
return -part.CFrame.LookVector
end
end

local function GetFacePos(part, normalId)
local facepos = part.CFrame * Vector3.FromNormalId(normalId)
if normalId == Enum.NormalId.Top or normalId == Enum.NormalId.Bottom then
return facepos.Y
elseif normalId == Enum.NormalId.Left or normalId == Enum.NormalId.Right then
return facepos.X
elseif normalId == Enum.NormalId.Front or normalId == Enum.NormalId.Back then
return facepos.Z
end
end

for _, part1 in pairs(parts) do
local randomColor1 = BrickColor.random()
for _, part2 in pairs(parts) do
if part1 ~= part2 then
local randomColor2 = BrickColor.random()
for _, face in pairs(faces) do
local part1Normal = GetNormal(part1, face)
local part2Normal = GetNormal(part2, face)
local part1Facepos = GetFacePos(part1, face)
local part2Facepos = GetFacePos(part2, face)
if part1Facepos == part2Facepos then
local surfaceGUI1 = Instance.new(“SurfaceGui”)
surfaceGUI1.Parent = part1
surfaceGUI1.Face = face

 local surfaceGUI2 = Instance.new("SurfaceGui")
 surfaceGUI2.Parent = part2
 surfaceGUI2.Face = face
 
 local frame1 = Instance.new("Frame")
 frame1.Parent = surfaceGUI1
 frame1.Size = UDim2.new(1,0,1,0)
 frame1.BackgroundColor = randomColor2
 
 local frame2 = Instance.new("Frame")
 frame2.Parent = surfaceGUI2
 frame2.Size = UDim2.new(1,0,1,0)
 frame2.BackgroundColor = randomColor2
end

end
end
end
end
]]

Not sure I get what this is, could you elaborate please?

I am the architec to the solution which is the solution whcih is the solution to your hence mentioned poroblem you stated earlier along the space time continuem.

I ended up trying something just kind of as a proof of concept:

It’s obviously not great (and its a bit laggy) but the way I did it is by essentially raycasting for each of these “pixels” on the screen out from the camera. Then I tracked the normals of all the raycasts, and compared each pixels detected normal to the ones around it. If the normal is different, it must mean that the surface is not flat and bends at that point. If so, it makes that pixel black (all other pixels are transparent so that you can see the actual game)

The theory is kinda there. I have to do some more testing to see what will bottleneck it (I assume it’s the raycasts, but I’ll make sure)

Another problem that I found is on things like stairs. Because the normals of the raycasts between two pixels when you face a staircase is the same, no outline is drawn. This could probably be fixed by also keeping track of the pixels distance from the camera (aka depth buffer).

The idea is hopefully to get a higher fidelity on the raycasts (I really hope that the GUI updating is the actual bottleneck…) by replacing this GUI method with something else.

1 Like

I think I’ve figured it out.

Take a look at stravant’s GapFill plugin.

Gonna try and reverse engineer this tomorrow.

1 Like

This is cool, very nice seeing that you made something this cool this quickly. I believe that I saw something like this before, can’t find the post, but remember the people working on it had been dealing with a lot of performance issues because the amount of ray casting; it was a while ago and their have been a lot of updates since then, but you’ll still be limited in resolution and how much you can do with it. Still cool tho. BTW, if your able to make this more detailed, consider making lines further away thicker or skinnier; Manifold Garden did this, thought it was pretty cool.

looks interesting, hopefully you find something useful in it.

Still working on it, but I found the actual code that does the edge finding.

It only finds the CLOSEST edge, and the way it does it is like this: you feed it a ray cast, and it gets the position and normal of the raycast result (which represents a plane in 3d space). It then explores outward on the plane with raycasts until it goes past the edge, then (I think) it moves down relative to the surface of the part we were hitting, and raycasts back towards the origin to get a position and normal of the next face (which is another plane in 3d space). It then calculates the ray that represents the line of intersection between the two planes we have, and then explores along that edge with raycasts (somehow, I don’t quite understand that right now.)