5/13/20–Worldmodel officially released! so there is probably no need for the method described here anymore!
i recently made a relatively “quick” set of posts here: Mouse.Hit Inside ViewPort Frames? about raycasting and finding Mouse.Hit/Target in viewport frames and i thought i’d make a post here to go more in depth or provide more info on how to actucally do it and while i am at it some more functionailty. However, keep in mind i won’t go too in depth into how the specific math works on all components in this tutorial, as i want to keep it “quite to the point” and even i am not fully confident as to explain all of it( but i will leave some resources below 
!)
For those who don’t know already there is an upcoming addition to viewport frames , the WorldModel class! that will add in some functional to raycast natively in viewport frames! making this post in the future completely obsolete, but hopefully this will still be useful to someone. I also enjoyed making it anyways.
Quick note: largest downside to the method described here is that you have to define some boundaries of parts that are not “simple” for intersection ( relatively easy though).
Ok, So firstly how do you go about getting the Target and hit position?
This in theory is quite simple: (Most Basic Steps)
- The first thing to do is to use a function to translate our Mouse Coordinates to World Coordinates without using rays
- Once we have this we need to raycast in the direction of our translated mouse position. Well actually we can’t do this ( kind of mentioned above) because raycasts aren’t simulated in viewport frames (for now), well specifically physics aren’t, which is what is needed to raycast ( at least in roblox) so what we need to do is “fake raycast”. We need an algorithm that will allow us to do some quick intersection tests. For the sake of “simplicity” and the efficiency ratio i am opting to use the Gilbert–Johnson–Keerthi Algorithm (GJK). Oh and because @IdiomicLanguage has a ready to use open-sourced module which is what i am using for his tutorial !
Technically that is it for simply Finding the Target but now what about finding the hit position well this one is a little less trivial. The way to do, or at least the way i choose to do it is to:
- Once we know which object has been hit First thing we need to do is Find the point of intersection of the part’s Bounding Box , oh and i forgot mention that GJK doesn’t normally give you the point of intersection in space and time without modifying it, but i won’t be doing that for now. instead i choose to find the point of intersection from it’s bounding box (AABB or OBB) by testing on all three axises.
- Once we found the intersection point, both
tFar(Farthest intersection away from our “ray origin” ) andtNear(Closet to our “Ray origin”) we need to Utilize GJK again to interpolate/translate a point fromtNeartotFarchecking if we have intersected with the object in question. When we have intersected we return the point that we were currently at (or the point just before impact), this will give us our hit position. The only downside is if you interpolate a point using a large increment small inaccuracies can occur. (more on that later)
That is basically it for Finding the Hit Position but this is all just text we need to look at how to do this in code.
Go to bottom for module/source
ScreenToWorldConversion
So First thing first is to find the world position of our mouse. i am using part of a method that is kind of described in a couple places but one that i originally found is here
Anyways, what we need to do is to convert our mouse position to what i like to call a local mouse position by subtracting our mouse position by our absolute position of our Viewport frame, Which goes something like this:
local function LocalPos(Pos,Viewport)
return Pos - Viewport.AbsolutePosition --- Return Position - vpf.AbsPos
end
Now we need to get the Viewport Frame’s Aspect ratio which can be defined as vpf.AbsSize.X/vpf.AbsSize.Y
Next, calculate the tangent of the camera’s Fov/2 so that we can multiply it by our aspect ratio and later fx.
local Tangent = math.tan(math.rad(Fov/2))
Before we mulitply our tangent by anything lets find fx and fy (Where Scale is a chosen number or 1) , which are going to be our mouse positions but from a value of -1 to 1 * Scale ( if Scale =2 values are -2 to 2) .
----Pos = mouse position
---Scale = depth or 1
local Scale = DepthParam or 1
local fx = ((2 * Scale) * (Pos.x /(Width-1)) -(Scale*1)) --x pos from -1 to 1
local fy = ((2 * Scale) * (Pos.y/(Height-1)) -(Scale*1))-- y pos from -1 to 1
Now it’s time to get “world mouse position” , For getting our X position, multiply the AspectRatio by Tangent and then by fx
local NX = AspectRatio * Tangent * fx
For Y multiply -Tangent by fy
local NY = (-Tangent * fy)
Finally for the Z axis just use -Scale
local NZ = -Scale
So now our new position to worldspace becomes:
local Pos = Vector3.new(NX, NY, NZ)
Ok so that was pretty much it for Getting the “World Mouse Position”, but there is two problems, this wont work for any camera that is rotated and in fact the position we calculated isn’t “rotated” either to reflect the “viewing fustrum”. To fix this we should multiply our camera’s CFrame by our position:
local Translatedcf = camera.CFrame * CFrame.new(Vector3.new(NX, NY, NZ))
Now to “rotate” our position we can simply use CFrame.new(pos, lookAt):
local FinalCf = CFrame.new( Translatedcf.Position, camera.CFrame.Position)
Then there we go we have Mouse position that is now in world space, our final funcion would/can look something like this:
local function ScreenToWorldSpace(Pos, viewport, camera, Depth, Gui_Inset)
Pos = LocalPos(Vector2.new(Pos.X, Pos.Y ), viewport) -- Make mouse position "local"
local Cam_Size = Vector2.new(viewport.AbsoluteSize.X , viewport.AbsoluteSize.Y - (Gui_Inset or 0))
local Height = Cam_Size.Y
local Width = Cam_Size.X
local AspectRatio = (Cam_Size.X/Cam_Size.Y)
local Cam_Pos = camera.CFrame.Position
local Scale =(Depth or 1)
local fov =math.rad(camera.FieldOfView)
local Tangent = math.tan((fov/2));
local fx = ((2 * Scale) * (Pos.x /(Width-1)) -(Scale*1))
local fy = ((2 * Scale) * (Pos.y/(Height-1)) -(Scale*1))
local NX = ((AspectRatio * Tangent * fx ))
local NY = (-Tangent * fy)
local NZ = -Scale
local Translatedcf = (camera.CFrame) * CFrame.new(Vector3.new(NX, NY, NZ)) -- rotate rel to camera
return CFrame.new( Translatedcf.p , camera.CFrame.p ) -- rotate to face camera
end
local pos = ScreenToWorldSpace(MousePos, ViewportFrame, Camera)
“Raycasting”
Now that we have that function all we need to do is “raycast” using GJK, where we need to do define a support function and can define a separation vector. i will not go too in depth with how and why this is needed especially since i am no master at the GJK algorithm but i’ll leave some resources below.
The support function given a dir (direction) will return the furthest point which is within our object, this will allow GJK to check if, in our case if a ray is intersecting. For now in this tutorial i am going to use two mainsupport functions one for spheres and one for parts (boxs) . Stole some of these from some of IdiomicLanguage’s Posts:
For a part the support function can be calculated using the dot product and it’s look/right/up vector …
local pPos, pI, pJ, pK
local function supportPart(dir)
return pPos
+ (pI:Dot(dir) > 0 and pI or -pI)
+ (pJ:Dot(dir) > 0 and pJ or -pJ)
+ (pK:Dot(dir) > 0 and pK or -pK)
end
Where pPos is the part’s position, pI is Part.CFrame.RightVector * (Part.Size.X/2) ,
pJ is Part.CFrame.UpVector * (Part.Size.Y/2)
pK is Part.CFrame.LookVector * (Part.Size.Z/2)
For a sphere the support function can just use the sphere radius and pos and the UnitVec of dir:
local pos, radius
local function SupportSphere(dir)
return pos + dir.Unit * radius
end
One Last Support function we most definitely need is the RaySupport function!
local rayStart, rayFinish
local function supportRay(dir)
if rayStart:Dot(dir) > rayFinish:Dot(dir) then
return rayStart
else
return rayFinish
end
end
where rayStart is the start position of our ray in our case the “Mouse’s World Position” and rayFinish is rayStart + dir * MaxLength
Lastly we could also use a separation/rejection vector function, Again stole this one specifically from this thread.
local function rejectionVector(point, start, dir)
local vStartToPoint = point - start
return vStartToPoint - vStartToPoint:Dot(dir) * dir
end
Now that we have all of that we can raycast, first we would check what type of part we are dealing with check (Sphere, Box etc.) then give GJK our specific support func and our raySupport func and our rejection vector to test if they are intersecting.
Using IdiomicLanguage’s module we can do this:
local function MagnitudeSquared(point, start, dir)
local vStartToPoint = point - start
local projectionMagnitude = vStartToPoint:Dot(dir)
return vStartToPoint:Dot(vStartToPoint) - projectionMagnitude * projectionMagnitude
end
The function above is going to help use just reduce the number of parts we are going to have to test, in our raycast function
...
local function SupportSphere(Part)
local pos = Part.Position
local s = Part.Size * 0.5
local radius = math.min(s.X, s.Y, s.Z)
return function(dir)
return pos + dir.Unit * radius
end
end
local function supportRay (rayStart, rayFinish)
return function (dir)
if rayStart:Dot(dir) > rayFinish:Dot(dir) then
return rayStart
else
return rayFinish
end
end
end
local function Raycast(start, dir, parts,maxDist, ReturnAll )
maxDist = maxDist or DefaultRayLength or 500
local finish = start + dir * maxDist
local Foundparts = {}
for _, part in ipairs(parts) do
local Mag = rejectionMagnitudeSquared(part.Position, start, dir)
if Mag < part.Size:Dot(part.Size) then --- if the part is worth it to continue
local SupportFunc = SupportSphere
local vec = rejectionVector(part.Position, start, dir) -- rejection vector
if GJK.intersection(SupportFunc(part) ,supportRay(start, finish),vec) then -- intersect?
print("intersection!!")
end
end
end
end
What about the Hit Position?
We need to now find our point of intersection of our AABB or OBB, normally calculating AABB is “cheaper” than calculating the OBB intersection so that is why we are checking if the part is un-rotated. i won’t go in depth like i said but what we need to do is first add all parts that we found in our RayCast function into a table (FoundParts)
if GJK.intersection(SupportFunc,supportRay,vec) then -- intersecting?
table.insert(FoundParts, {part, SupportFunc})
end
then go through all parts and find their points of intersection on AABB or OBB then translate them to the actual points of intersection:
ToFindAABBIntersection
AABB or Axis-Aligned Bounding Box doesn’t rotate with the object it stays axis aligned so if you were to use the code below for a rotated object you might notice that the point of intersection is incorrect
(Hard to tell but the purple and blue parts are the “intersections”)
To find the point of intersection for and AABB we can do this:
local function AABBLineIntersection(AABB, S, E)
local StoE = E - S
local P = AABB.Position
local Size = AABB.Size/2
local min = P - Size
local max = P + Size
local Bmin = min-S
local Bmax = max - S
local Near = -math.huge
local Far = math.huge
local intersect = false
local p1
local p2
for _, Axis in ipairs(Enum.Axis:GetEnumItems()) do
if StoE[Axis.Name] == 0 then
if (Bmin[Axis.Name] > 0 ) or (Bmax[Axis.Name] < 0) then
return p1, p2
end
else
local t1 = Bmin[Axis.Name]/StoE[Axis.Name]
local t2 = Bmax[Axis.Name]/StoE[Axis.Name]
local tmin = math.min(t1, t2)
local tmax = math.max(t1,t2)
if tmin > Near then
Near = tmin
end
if tmax < Far then
Far = tmax
end
if Near > Far or Far < 0 then
return intersect, p1, p2
end
end
end
if Near >= 0 and Near <= 1 then
p1 = S + StoE * Near
end
if Far >= 0 and Far <= 1 then
p2 = S + StoE * Far
end
if p1 and p2 then
intersect = true
end
return intersect, p1, p2
end
ToFindOBBIntersection
OBB or Oriented Bounding Box essentially means that the bounding box moves with the rotated object, in other words it can rotate. This time unlike for AABBs our point of intersection is correct.
local function Scale(vec, n)
return Vector3.new(vec.X * n ,vec.Y * n, vec.Z * n )
end
local function OBBLineIntersection(OBB, Start, End )
local Dir = End - Start
local Delta = OBB.Position - Start
local CF = OBB.CFrame
local Bmax = CF:PointToObjectSpace(CF * (OBB.Size/2))
local Bmin = CF:PointToObjectSpace(CF * (-OBB.Size/2))
local CurrAxis
local tMin = -math.huge
local tMax = math.huge
local epsilon= 0.00001
local p,nomLen,denomLen,tmp,min,max;
for _, Axis in ipairs(Enum.Axis:GetEnumItems()) do
if Axis == Enum.Axis.X then
CurrAxis = CF.RightVector
elseif Axis == Enum.Axis.Y then
CurrAxis = CF.UpVector
else
CurrAxis = CF.LookVector
end
nomLen = Delta:Dot(CurrAxis)
denomLen = CurrAxis:Dot(Dir)
if math.abs(denomLen) > epsilon then
min = (nomLen + Bmin[Axis.Name] )/denomLen
max = (nomLen + Bmax[Axis.Name] )/denomLen
if min > max then
tmp = min; min = max; max = tmp
end
if min > tMin then
tMin = min
end
if max < tMax then
tMax = max
end
if tMax < tMin then
return false
end
elseif ((-nomLen + Bmin[Axis.Name] ) > 0) or (-nomLen +Bmax[Axis.Name] <0) then
return false
end
end
local FirstIntersect = Scale(Dir, tMin ) + Start
local LeavingIntersect = Scale(Dir, tMax ) + Start
return true, FirstIntersect, LeavingIntersect
end
local funciton HitFunction(RayOrigin, Rayend, dir , Parts)
local intersections = {} --Blank Table to hold our intersections vectors
for _, PartInfo in ipairs(FoundParts) do
local part = PartInfo[1]
local BBIntersect, Start, End
local Support = PartInfo[2] --Support function
local Length = 0 ---Start Length...
local intersection = false
if part.Orientation == Vector3.new() then --- we can check if the part is unrotated if so calc AABB else calc OBB
---- calc AABB intersection
BBIntersect, Start, End = AABBLineIntersection(part,RayOrigin, Rayend)
else
---- calc OBB intersection
BBIntersect, Start, End = OBBLineIntersection(part,RayOrigin, Rayend)
end
.....
The Next part is to translate our Start Intersection Start to our End Intersection End, for this we need another support func (similar to the part support function):
local function SupportPoint(pointvec)
return function(dir)
local cf = CFrame.new(pointvec) - pointvec
local SP_R, SP_U, SP_L = cf.RightVector , cf.UpVector , cf.LookVector
return pointvec
+ (SP_R:Dot(dir) > 0 and SP_R or -SP_R)
+ (SP_U:Dot(dir) > 0 and SP_U or -SP_U)
+ (SP_L:Dot(dir) > 0 and SP_L or -SP_L)
end
end
SP_R for example is just cf.RightVector without the need multiply a size, because there is not size to a single vector!
Continuing our Hit Function:
if BBIntersect then -- if bounding box intersects
while not intersection do -- while pointvec isnt intersecting...
local pointvec = Start + dir * Length --- move point near end point
Length = Length + 0.05 -- add our increment
if Length >= (Start - End).magnitude then --- if we have not passed the end point
intersections[part] = Vector3.new(pointvec.X, pointvec.Y,pointvec.Z) --record data
intersection = true --- exit loop
end
if GJK.intersection(partType(part), SupportPoint(pointvec), Vector3.new(0,1,0) ) then --- if intersection
intersections[part] = Vector3.new(pointvec.X, pointvec.Y,pointvec.Z) --- record data
intersection = true --- exit loop
end
end
You might have noticed something different with our last portion of our Hit function, we actually only need a up vector (0,1,0) for our rejection vector instead of what we were doing before.
where i did Length = Length + .05 this is what i was talking about increments, the higher it is the more chance it is for inaccuracies but usually it leads to better performance. i found that 0.05 was Good because that is the smallest size a part can go on all axises.
Lastly we need to just do some sorting and find our best candidate for our target by finding the part with the lowest magnitude from the origin/start of the ray using its hit position that we calculated earlier:
local Best
local BestScore = math.huge --- INFINITY
for _, v in pairs(intersections) do
local mag = (RayOrigin - v ).Magnitude
if (mag < BestScore) then
BestScore = mag
Best =_
Hit = v
end
----------Best is now our Mouse.Target
---Hit is now our Mouse.Hit
That’s it!
If everything turned out right…
(Using the model rotation thing from this post)
if you notice the artifact in the video were the red part clips the humanoid there was a small code error when I was recording the video silly me 
it should be fixed as of now though
Here is everything put together , along with some extra functions and stuff…oh and the hitboxes are a bit off (they are all “boxes”) in the provided example that’s why it might not return the correct part but you could easily adjust that by adding a support function, using a another one, or using the SupportPolygon function inside the module.
ViewportFrameRayCasting.rbxl (40.5 KB)
Conclusion:
i think this is overall an ok way to do raycasting in viewport frames and actually isn’t too “expensive”, i was running a solid 60fps while “raycasting” inside a heartbeat and renderstepped, there however might be optimizations that could be done. if you have any questions or improvements or something i missed something please let me know! Also if someone would like to explain the math a little more in detail that would be great too
This is my first tutorial btw
Helpful Resources
Picking with custom Ray-OBB function
Visualizing the GJK Collision detection algorithm — Harold Serrano - Game Engine Developer
WebGL 2.0 : 046 : Ray intersects Bounding Box (AABB) - YouTube
WebGL 2.0 : 047 : Ray intersects Bounding Box (OBB) - YouTube
3d - Intersection of line segment with axis-aligned box in C# - Stack Overflow
c++ - OpenGL - Mouse coordinates to Space coordinates - Stack Overflow
if you are ever looking for a great alternative for mouse.hit/target in viewportframes other than the method described in this post, Onogork has a great module.
