How to find Mouse Target/Hit in Viewport Frames

5/13/20–Worldmodel officially released! so there is probably no need for the method described here anymore!

i recently made a relatively “quick” set of posts here: Mouse.Hit Inside ViewPort Frames? about raycasting and finding Mouse.Hit/Target in viewport frames and i thought i’d make a post here to go more in depth or provide more info on how to actucally do it and while i am at it some more functionailty. However, keep in mind i won’t go too in depth into how the specific math works on all components in this tutorial, as i want to keep it “quite to the point” and even i am not fully confident as to explain all of it( but i will leave some resources below :grinning:!)


For those who don’t know already there is an upcoming addition to viewport frames , the WorldModel class! that will add in some functional to raycast natively in viewport frames! making this post in the future completely obsolete, but hopefully this will still be useful to someone. I also enjoyed making it anyways.


Quick note: largest downside to the method described here is that you have to define some boundaries of parts that are not “simple” for intersection ( relatively easy though).


Ok, So firstly how do you go about getting the Target and hit position?

This in theory is quite simple: (Most Basic Steps)

  1. The first thing to do is to use a function to translate our Mouse Coordinates to World Coordinates without using rays
  2. Once we have this we need to raycast in the direction of our translated mouse position. Well actually we can’t do this ( kind of mentioned above) because raycasts aren’t simulated in viewport frames (for now), well specifically physics aren’t, which is what is needed to raycast ( at least in roblox) so what we need to do is “fake raycast”. We need an algorithm that will allow us to do some quick intersection tests. For the sake of “simplicity” and the efficiency ratio i am opting to use the Gilbert–Johnson–Keerthi Algorithm (GJK). Oh and because @IdiomicLanguage has a ready to use open-sourced module which is what i am using for his tutorial :grinning:!

Technically that is it for simply Finding the Target but now what about finding the hit position well this one is a little less trivial. The way to do, or at least the way i choose to do it is to:

  1. Once we know which object has been hit First thing we need to do is Find the point of intersection of the part’s Bounding Box , oh and i forgot mention that GJK doesn’t normally give you the point of intersection in space and time without modifying it, but i won’t be doing that for now. instead i choose to find the point of intersection from it’s bounding box (AABB or OBB) by testing on all three axises.
  2. Once we found the intersection point, both tFar (Farthest intersection away from our “ray origin” ) and tNear(Closet to our “Ray origin”) we need to Utilize GJK again to interpolate/translate a point from tNear to tFar checking if we have intersected with the object in question. When we have intersected we return the point that we were currently at (or the point just before impact), this will give us our hit position. The only downside is if you interpolate a point using a large increment small inaccuracies can occur. (more on that later)

That is basically it for Finding the Hit Position but this is all just text we need to look at how to do this in code.


Go to bottom for module/source

ScreenToWorldConversion

So First thing first is to find the world position of our mouse. i am using part of a method that is kind of described in a couple places but one that i originally found is here

Anyways, what we need to do is to convert our mouse position to what i like to call a local mouse position by subtracting our mouse position by our absolute position of our Viewport frame, Which goes something like this:

local function LocalPos(Pos,Viewport)
	return Pos - Viewport.AbsolutePosition --- Return Position - vpf.AbsPos
end

Now we need to get the Viewport Frame’s Aspect ratio which can be defined as vpf.AbsSize.X/vpf.AbsSize.Y
Next, calculate the tangent of the camera’s Fov/2 so that we can multiply it by our aspect ratio and later fx.

local Tangent = math.tan(math.rad(Fov/2)) 

Before we mulitply our tangent by anything lets find fx and fy (Where Scale is a chosen number or 1) , which are going to be our mouse positions but from a value of -1 to 1 * Scale ( if Scale =2 values are -2 to 2) .

----Pos = mouse position
---Scale = depth or 1
 local Scale = DepthParam or 1
 local fx = ((2 * Scale) * (Pos.x /(Width-1)) -(Scale*1)) --x pos from -1 to 1 
 local fy = ((2 * Scale) * (Pos.y/(Height-1)) -(Scale*1))-- y pos from -1 to 1

Now it’s time to get “world mouse position” , For getting our X position, multiply the AspectRatio by Tangent and then by fx

local NX =  AspectRatio * Tangent * fx 

For Y multiply -Tangent by fy

local NY = (-Tangent * fy)

Finally for the Z axis just use -Scale

local NZ = -Scale 

So now our new position to worldspace becomes:

local Pos =  Vector3.new(NX, NY, NZ)

Ok so that was pretty much it for Getting the “World Mouse Position”, but there is two problems, this wont work for any camera that is rotated and in fact the position we calculated isn’t “rotated” either to reflect the “viewing fustrum”. To fix this we should multiply our camera’s CFrame by our position:

 local Translatedcf = camera.CFrame * CFrame.new(Vector3.new(NX, NY, NZ))

Now to “rotate” our position we can simply use CFrame.new(pos, lookAt):

local FinalCf =  CFrame.new( Translatedcf.Position, camera.CFrame.Position) 

Then there we go we have Mouse position that is now in world space, our final funcion would/can look something like this:

local function ScreenToWorldSpace(Pos, viewport, camera, Depth, Gui_Inset)
 Pos =  LocalPos(Vector2.new(Pos.X, Pos.Y ), viewport) -- Make mouse position "local"
 local Cam_Size = Vector2.new(viewport.AbsoluteSize.X , viewport.AbsoluteSize.Y - (Gui_Inset or 0))
 local Height = Cam_Size.Y
 local Width = Cam_Size.X	
 local AspectRatio = (Cam_Size.X/Cam_Size.Y)
 local Cam_Pos = camera.CFrame.Position 
 local Scale =(Depth or 1) 
 local fov  =math.rad(camera.FieldOfView)
 local Tangent = math.tan((fov/2));
 local fx = ((2 * Scale) * (Pos.x /(Width-1)) -(Scale*1))
 local fy = ((2 * Scale) * (Pos.y/(Height-1)) -(Scale*1))
 local NX = ((AspectRatio * Tangent * fx ))
 local NY = (-Tangent * fy)
local NZ = -Scale 
local Translatedcf = (camera.CFrame) * CFrame.new(Vector3.new(NX, NY, NZ))  -- rotate rel to camera
    return CFrame.new( Translatedcf.p , camera.CFrame.p  ) -- rotate to face camera
end 

local pos = ScreenToWorldSpace(MousePos, ViewportFrame, Camera)

“Raycasting”

Now that we have that function all we need to do is “raycast” using GJK, where we need to do define a support function and can define a separation vector. i will not go too in depth with how and why this is needed especially since i am no master at the GJK algorithm but i’ll leave some resources below.

The support function given a dir (direction) will return the furthest point which is within our object, this will allow GJK to check if, in our case if a ray is intersecting. For now in this tutorial i am going to use two mainsupport functions one for spheres and one for parts (boxs) . Stole some of these from some of IdiomicLanguage’s Posts:

For a part the support function can be calculated using the dot product and it’s look/right/up vector …

local pPos, pI, pJ, pK
local function supportPart(dir)
	return pPos
	+ (pI:Dot(dir) > 0 and pI or -pI)
	+ (pJ:Dot(dir) > 0 and pJ or -pJ)
	+ (pK:Dot(dir) > 0 and pK or -pK)
end

Where pPos is the part’s position, pI is Part.CFrame.RightVector * (Part.Size.X/2) ,
pJ is Part.CFrame.UpVector * (Part.Size.Y/2)
pK is Part.CFrame.LookVector * (Part.Size.Z/2)

For a sphere the support function can just use the sphere radius and pos and the UnitVec of dir:

local pos,  radius
local function SupportSphere(dir)
	return pos + dir.Unit * radius
end

One Last Support function we most definitely need is the RaySupport function!

local rayStart, rayFinish
local function supportRay(dir)
	if rayStart:Dot(dir) > rayFinish:Dot(dir) then
		return rayStart
	else
		return rayFinish
	end
end

where rayStart is the start position of our ray in our case the “Mouse’s World Position” and rayFinish is rayStart + dir * MaxLength

Lastly we could also use a separation/rejection vector function, Again stole this one specifically from this thread.

local function rejectionVector(point, start, dir)
	local vStartToPoint = point - start
	return vStartToPoint - vStartToPoint:Dot(dir) * dir
end

Now that we have all of that we can raycast, first we would check what type of part we are dealing with check (Sphere, Box etc.) then give GJK our specific support func and our raySupport func and our rejection vector to test if they are intersecting.

Using IdiomicLanguage’s module we can do this:

local function MagnitudeSquared(point, start, dir)
	local vStartToPoint = point - start
	local projectionMagnitude = vStartToPoint:Dot(dir)
	return vStartToPoint:Dot(vStartToPoint) - projectionMagnitude * projectionMagnitude
end

The function above is going to help use just reduce the number of parts we are going to have to test, in our raycast function

...


local function SupportSphere(Part)
local pos = Part.Position
local s = Part.Size * 0.5
local radius = math.min(s.X, s.Y, s.Z)
  return function(dir)
	return pos + dir.Unit * radius
   end
end


local function supportRay (rayStart, rayFinish)
 return function (dir)
	if rayStart:Dot(dir) > rayFinish:Dot(dir) then
		return rayStart
	else
		return rayFinish
	end
  end
end


local function Raycast(start, dir,  parts,maxDist, ReturnAll )
maxDist = maxDist or DefaultRayLength or 500
  local finish = start + dir * maxDist 
   local Foundparts = {}
    for _, part in ipairs(parts) do
	local Mag = rejectionMagnitudeSquared(part.Position, start, dir)
     if Mag < part.Size:Dot(part.Size) then --- if the part is worth it to continue 
             local SupportFunc =  SupportSphere
			local  vec = rejectionVector(part.Position, start, dir) -- rejection vector
	     	 if GJK.intersection(SupportFunc(part) ,supportRay(start, finish),vec) then -- intersect?
		print("intersection!!")
     end
    end
  end
end

What about the Hit Position?


We need to now find our point of intersection of our AABB or OBB, normally calculating AABB is “cheaper” than calculating the OBB intersection so that is why we are checking if the part is un-rotated. i won’t go in depth like i said but what we need to do is first add all parts that we found in our RayCast function into a table (FoundParts)

   if GJK.intersection(SupportFunc,supportRay,vec) then -- intersecting?
         table.insert(FoundParts, {part, SupportFunc})
    end

then go through all parts and find their points of intersection on AABB or OBB then translate them to the actual points of intersection:

ToFindAABBIntersection

AABB or Axis-Aligned Bounding Box doesn’t rotate with the object it stays axis aligned so if you were to use the code below for a rotated object you might notice that the point of intersection is incorrect


(Hard to tell but the purple and blue parts are the “intersections”)

To find the point of intersection for and AABB we can do this:

local function AABBLineIntersection(AABB, S, E)
local StoE = E - S
local P = AABB.Position
local Size = AABB.Size/2
local min = P - Size
local max = P + Size
local Bmin = min-S
local Bmax = max - S
local Near = -math.huge
local Far = math.huge
local intersect = false
local p1
local p2
  for _, Axis in ipairs(Enum.Axis:GetEnumItems()) do
  if StoE[Axis.Name] == 0 then
    if (Bmin[Axis.Name] > 0 ) or (Bmax[Axis.Name] < 0) then
	    return p1, p2
	  end
   else
  local t1 = Bmin[Axis.Name]/StoE[Axis.Name]
  local t2 = Bmax[Axis.Name]/StoE[Axis.Name]
  local tmin = math.min(t1, t2)
  local tmax = math.max(t1,t2)
  if tmin > Near then
   Near = tmin
end
 if tmax < Far then
	Far = tmax
end
if Near > Far or Far < 0 then
  return  intersect, p1, p2
	 end	
   end
end
  	if Near >= 0 and Near <= 1 then
     	p1 =  S + StoE * Near 
	end
	if Far >= 0 and Far <= 1 then
	p2 =  S + StoE * Far
   end
  if p1 and p2 then
	  intersect = true
end

    return  intersect, p1, p2
end


ToFindOBBIntersection

OBB or Oriented Bounding Box essentially means that the bounding box moves with the rotated object, in other words it can rotate. This time unlike for AABBs our point of intersection is correct.


local function Scale(vec, n)
   return Vector3.new(vec.X * n ,vec.Y * n, vec.Z * n )
end

local function OBBLineIntersection(OBB, Start, End )
 local Dir = End - Start
 local Delta = OBB.Position - Start
 local CF = OBB.CFrame
 local Bmax =  CF:PointToObjectSpace(CF * (OBB.Size/2)) 
 local Bmin = CF:PointToObjectSpace(CF * (-OBB.Size/2)) 
 local CurrAxis 
 local tMin = -math.huge
 local tMax = math.huge	
 local epsilon= 0.00001
  local p,nomLen,denomLen,tmp,min,max;
     for _, Axis in ipairs(Enum.Axis:GetEnumItems()) do
	  if Axis == Enum.Axis.X then
         CurrAxis  = CF.RightVector
		elseif Axis == Enum.Axis.Y then
	    CurrAxis  = CF.UpVector
		else
       CurrAxis = CF.LookVector
	end
	nomLen = Delta:Dot(CurrAxis)
      denomLen = CurrAxis:Dot(Dir)
	 if math.abs(denomLen) >  epsilon  then
		min = (nomLen + Bmin[Axis.Name] )/denomLen
		max = (nomLen + Bmax[Axis.Name]  )/denomLen
		if min > max then
			tmp = min; min = max; max = tmp
		end
	     if min > tMin then
          tMin = min
	   end
	        if max < tMax then
		    tMax = max
	        end
	       if tMax < tMin then
		return false
	     end
      	elseif  ((-nomLen + Bmin[Axis.Name] ) > 0) or (-nomLen +Bmax[Axis.Name]  <0) then
		return false
      end
  end
local FirstIntersect = Scale(Dir, tMin ) + Start
local LeavingIntersect = Scale(Dir, tMax ) + Start
    return true, FirstIntersect, LeavingIntersect
end
local funciton HitFunction(RayOrigin, Rayend, dir , Parts)
   local intersections = {} --Blank Table to hold our intersections vectors
   for _, PartInfo in ipairs(FoundParts) do
    local part = PartInfo[1]
    local BBIntersect, Start, End 
    local Support = PartInfo[2] --Support function
    local Length = 0 ---Start Length...
    local intersection = false 
	if part.Orientation == Vector3.new() then --- we can check if the part is unrotated if so calc AABB else calc OBB
      ---- calc AABB intersection
      	 BBIntersect, Start, End  =  AABBLineIntersection(part,RayOrigin,  Rayend)
		else
      ---- calc OBB intersection
	  BBIntersect, Start, End  = OBBLineIntersection(part,RayOrigin,  Rayend)
   end
.....

The Next part is to translate our Start Intersection Start to our End Intersection End, for this we need another support func (similar to the part support function):

local function SupportPoint(pointvec)
 return function(dir)
 local cf = CFrame.new(pointvec) - pointvec
  local  SP_R, SP_U, SP_L = cf.RightVector , cf.UpVector , cf.LookVector 
	return pointvec
	+ (SP_R:Dot(dir) > 0 and SP_R or -SP_R)
	+ (SP_U:Dot(dir) > 0 and SP_U or -SP_U)
	+ (SP_L:Dot(dir) > 0 and SP_L or -SP_L)
   end
end

SP_R for example is just cf.RightVector without the need multiply a size, because there is not size to a single vector!

Continuing our Hit Function:

	 if  BBIntersect then -- if bounding box intersects
	    while  not intersection do -- while pointvec isnt intersecting...
 	       local pointvec = Start + dir * Length --- move point near end point
          Length = Length +  0.05 -- add our increment
          if Length >= (Start - End).magnitude then --- if we have not passed the end point
	         intersections[part] = Vector3.new(pointvec.X, pointvec.Y,pointvec.Z) --record data
             intersection = true --- exit loop
	      end
           if GJK.intersection(partType(part), SupportPoint(pointvec), Vector3.new(0,1,0) )   then --- if intersection 
	         intersections[part] =  Vector3.new(pointvec.X, pointvec.Y,pointvec.Z) --- record data
	      intersection = true --- exit loop
	  end
 end

You might have noticed something different with our last portion of our Hit function, we actually only need a up vector (0,1,0) for our rejection vector instead of what we were doing before.

where i did Length = Length + .05 this is what i was talking about increments, the higher it is the more chance it is for inaccuracies but usually it leads to better performance. i found that 0.05 was Good because that is the smallest size a part can go on all axises.

Lastly we need to just do some sorting and find our best candidate for our target by finding the part with the lowest magnitude from the origin/start of the ray using its hit position that we calculated earlier:

local Best
local BestScore = math.huge --- INFINITY
 for _, v in  pairs(intersections) do
local mag = (RayOrigin - v ).Magnitude 
	if (mag  < BestScore) then
		BestScore = mag
	    Best  =_
	  Hit = v
	end
----------Best is now our Mouse.Target
---Hit is now our Mouse.Hit


That’s it!

If everything turned out right…
(Using the model rotation thing from this post)

if you notice the artifact in the video were the red part clips the humanoid there was a small code error when I was recording the video silly me :man_facepalming: it should be fixed as of now though


Here is everything put together , along with some extra functions and stuff…oh and the hitboxes are a bit off (they are all “boxes”) in the provided example that’s why it might not return the correct part but you could easily adjust that by adding a support function, using a another one, or using the SupportPolygon function inside the module.
ViewportFrameRayCasting.rbxl (40.5 KB)


Conclusion:

i think this is overall an ok way to do raycasting in viewport frames and actually isn’t too “expensive”, i was running a solid 60fps while “raycasting” inside a heartbeat and renderstepped, there however might be optimizations that could be done. if you have any questions or improvements or something i missed something please let me know! Also if someone would like to explain the math a little more in detail that would be great too :grinning:

This is my first tutorial btw

Helpful Resources

Picking with custom Ray-OBB function

A Minimal Ray-Tracer: Rendering Simple Shapes (Sphere, Cube, Disk, Plane, etc.) (Ray-Box Intersection)

Visualizing the GJK Collision detection algorithm — Harold Serrano - Game Engine Developer

WebGL 2.0 : 046 : Ray intersects Bounding Box (AABB) - YouTube

WebGL 2.0 : 047 : Ray intersects Bounding Box (OBB) - YouTube

3d - Intersection of line segment with axis-aligned box in C# - Stack Overflow

c++ - OpenGL - Mouse coordinates to Space coordinates - Stack Overflow

if you are ever looking for a great alternative for mouse.hit/target in viewportframes other than the method described in this post, Onogork has a great module.


33 Likes

That’s actually… really useful.

I’m was going to wait for the release of WorldModel so I can make a click-y gun editor thing but I have this, kudos to you friend.

4 Likes

That’s dope dude, you look like you spent a long time doing this!

4 Likes

:sunglasses:

1 Like