Converting GUI Object to World Space

alphajpeg · April 8, 2022, 9:28am

Hello, I’m attempting to take a GUI Object’s AbsoluteSize and project it into the 3D space in-front of the camera. In simpler terms, I’m trying to make a brick the same size as a GUI Object, in-front of the camera.

My current approach to this was to to take the AbsoluteSize and make the following 2 points:

absoluteSize = 20, 10

pointA = 0, 0 --{0, 0}
pointB = absoluteSize.X, absoluteSize.Y --{20, 10}

which returns us something that looks like so:

Next, take our 2 points and use the ScreenPointToRay() function to project our 2 points into the 3d space. Once done, calculate the magnitude between pointA and pointB to get the size, like so:

worldPointA = camera:ScreenPointToRay(pointA.X, pointA.Y, 0.1).Origin
worldPointB = camera:ScreenPointToRay(pointB.X, pointB.Y, 0.1).Origin

x, = calculateMagnitude(worldPointA.X, worldPointB.X)
y, = calculateMagnitude(worldPointA.Y, worldPointB.Y)
--calculateMagnitude() = math.sqrt(a^2) + math.sqrt(b^2)

size = x, y, 0

My results were not as expected though, I’m probably missing or even over complicating something. Any help would be appericated, thanks.

just realized the amount of useless variables/code I have, but the idea still stands the same.

Lielmaster · April 8, 2022, 10:02am

Im assuming those two values are inside a Vector2? If not then look at this

a, b = 1, 2
print(a) -- 1
print(b) -- 2
c = 5,6
print(c) -- 5

Anyways, I think you know about that.

Is the position always at 0, 0? If not, then you still have to add the position to pointB.

Aside all of that, here is the main problem.

You are just taking the X and Y coordinates in a 3D space. You forgot to include Z.

alphajpeg · April 8, 2022, 10:05am

Yes the a and b variables are vector2’s. Why does the z coordinate matter here though? I’m attempting to get the x, y for the part, z axis won’t matter due to the part being placed in-front of the camera either way. etc 10, 10, 0.1

z axis is set to 0 either way.

ill make a diagram to help explain further in a second

Lielmaster · April 8, 2022, 10:09am

GUIs are in a 2D space. They can be in the X, and Y. And it is always in your screen, so no matter where you go, the X, and Y is still the same
But the camera is in a 3D space. The camera can be anywhere in the X, Y, and Z. What if you moved forward? then the Z axis will change. Since you only took the X and Y, there is no “Forward” axis.
What if you rotated your camera too? All axis changes, including Z.

alphajpeg · April 8, 2022, 10:17am

But ScreenPointToRay() returns a x, y coordinate to a vector3 infront of the camera, with the third parameter being distance/depth from the camera. I still don’t understand why the Z axis is required or involved anywhere here.

I wrote some code earlier that fills the screen with a part perfectly, with no use of the Z axis. Could that be used at all here?

Lielmaster · April 8, 2022, 10:21am

Hold on, I’m drawing something, to illustrate what I mean.

I’m assuming you never rotated your camera since Z is still part of the coordinates, but for some reason worked without it.

Lielmaster · April 8, 2022, 10:36am

Let’s say this is what’s happening.

This is what you see with points

Now let’s do the function. And remove the camera visual too.

If you included the Z axis, the points would have stayed where they were. Remember, that is where the camera is. Now look at the “without Z”. They are just staying in the XY plane because there is no Z or “forward” axis. Meaning, if you have moved/rotated your camera, the points will never go “forward”.

Imagine you are directly facing in the X Axis. There would be no difference in the X axis since all points are in the same X coordinate. Thus you will use the Z axis.

Hopefully that made sense.

alphajpeg · April 8, 2022, 10:42am

I see, thanks for your explanations. If you don’t mind, how would I go about implementing the Z axis?

Lielmaster · April 8, 2022, 10:44am

Well, Vector2 and Vector3 has their .Magnitude value so you may use that for the size instead.

size = (worldPointA - worldPointB).Magnitude

Lielmaster · April 8, 2022, 10:48am

Oh wait, I think I’m dumb. Magnitude formula is math.sqrt(a^2 + b^2). But in your code, you said math.sqrt(a^2) + math.sqrt(b^2). The math.sqrt just cancels the ^2. And also, you should subtract b from a before taking their magnitude, since magnitude is used for vectors.

alphajpeg · April 8, 2022, 11:07am

I’m so sorry, I’m still struggling on how to implement the Z axis with my current code. You most likely have the solution, I’m just too dumb to understand.

Lielmaster · April 8, 2022, 11:13am

If you just need this, then you are good with

magnitude = (worldPointA - worldPointB).Magnitude

You don’t have to do anything since the code already takes the magnitude of all axis.

alphajpeg · April 8, 2022, 11:14am

I’m experiencing this weird effect which is that the y axis size changes when the camera moves up and down. Other than that the x axis seems to be working fine.

TopBagon · April 8, 2022, 2:05pm

the image you have is actually pretty close to what you should do but the way you’re doing it is off.

notice that there are 2 triangles in the image, using mastermind EgoMoose’s article about triangles we can draw a quad using 2 triangles. here’s the main update code

local function updatePosition()
	local depth = 1 - 0.05 * frame.ZIndex
	local topLeft, bottomRight = frame.AbsolutePosition, frame.AbsolutePosition + frame.AbsoluteSize
	local topRight, bottomLeft = Vector2.new(bottomRight.X, topLeft.Y), Vector2.new(topLeft.X, bottomRight.Y)
	
	DrawQuad(
		camera:ScreenPointToRay(topLeft.X, topLeft.Y, depth).Origin, 
		camera:ScreenPointToRay(topRight.X, topRight.Y, depth).Origin, 
		camera:ScreenPointToRay(bottomLeft.X, bottomLeft.Y, depth).Origin, 
		camera:ScreenPointToRay(bottomRight.X, bottomRight.Y, depth).Origin
	)
end

here’s a place file
quad.rbxl (36.6 KB)

P.S if you wanna use that part as a canvas I'd recommend checking out Module3D instead

alphajpeg · April 10, 2022, 2:40am

I’ve made some advancements with this topic and I would consider myself pretty close, just have some inconsistent results with sizing and positioning.

If anyone would like to attempt this topic here is the test file:
(I’ve commented majority of the code for you.)
(TextLabel is set to not visible so you can see the part.)
3D UI Test.rbxl (40.0 KB)

FrancklinDay · January 24, 2024, 12:12am

very late bump but currently facing the same exact issue. thank you for providing your code sample. ive managed to fix this.

Looking at your code you’ve done everything correct except for the cornerCFrame part. you’ve calculated the corner according to the center of the part rather then following the same approach as a ui (which is the top left). to fix this you simply gotta add the text label size to the corner and divide it by 2 which fixes the X axis in the process (CFrame.new((textLabel.AbsoluteSize.X * screenRatio.X)/2). we also do the same thing to the Y axis.

Old problem. fixed on the edit

Now the problem ive faced right now which ive partially fixed is the Y axis. for some reason it seems like using the “uiPosition.Y” offsets it the more you modify the screen resolution for the y axis (if you attempt to test between mobile and computer screen. the y axis offsets pretty drastically) despite the fact that the corner alignment is perfect and doesn’t have any offset when testing it.

The work around for this is to use the Y size of the text label and transforming the the text label’s size to pure scale only (doesn’t seem to work if you use offset). then just add an offset to the cornercframe’s Y to align it with the ui. this seems to work perfectly on all kinds of resolutions.

I hope someone could provide an explanation for the Y axis problem.

EDIT:
Figured out whats going on for the y axis. its because the text label is calculating its absolute size/pos in accordance to its parent screen gui which takes into account of things like the gui inset. hense why the offset is happening. to fix this we have to calculate our own absolute position and size. but this time in accordance to the actual viewport extracted off the camera.

this is an example of how it looks like for a text label :

local absoluteSize = Vector2.new(
		textLabel.Size.X.Scale * viewportSize.X + textLabel.Size.X.Offset,
		textLabel.Size.Y.Scale * viewportSize.Y+ textLabel.Size.Y.Offset
)

local absolutePosition = Vector2.new(
		textLabel.Position.X.Scale * viewportSize.X + textLabel.Position.X.Offset,
		textLabel.Position.Y.Scale * viewportSize.Y + textLabel.Position.Y.Offset
)

And now from here. we just perform the same exact operation we’ve done to the width. which now becomes :
-(absoluteSize.Y * screenRatio.Y)/2

This is the final version. didn’t have enough time to organize the code sadly but hope this is understandeable enough :

local RunService = game:GetService("RunService")

local camera = game.Workspace.CurrentCamera
local part = game.Workspace.Part
local textLabel = script.Parent.Frame


local function viewportToStuds(viewportSize, depth, absoluteSize)
	local aspectRatio = viewportSize.X / viewportSize.Y --Get screen aspect ratio (1920 / 1080)
	
	local heightFactor = math.tan(math.rad(camera.FieldOfView)/2) --Get height of screen from camera
	local widthFactor = aspectRatio * heightFactor --Use height to also get the width

	
	local screenSize = Vector2.new(-2 * widthFactor * -depth, -2 * heightFactor * -depth) --Get size of the screen
	local screenRatio = Vector2.new(screenSize.X / viewportSize.X, screenSize.Y / viewportSize.Y) --The ratio for pixel to studs
	local screenCFrame = camera.CFrame * CFrame.new(0, 0, -depth) --Get where the screen is in the world

	local cornerCFrame = 
		screenCFrame 
		* CFrame.new(-screenSize.X/2, screenSize.Y/2, 0) 
		* CFrame.new((absoluteSize.X * screenRatio.X)/2, -(absoluteSize.Y * screenRatio.Y)/2, 0)
	--Get the corner of the screen
	--part.CFrame = cornerCFrame
	
	return screenSize, screenRatio, cornerCFrame
end

RunService.RenderStepped:Connect(function()
	local viewportSize = camera.ViewportSize
	
	local absoluteSize = Vector2.new(
		textLabel.Size.X.Scale * viewportSize.X + textLabel.Size.X.Offset,
		textLabel.Size.Y.Scale * viewportSize.Y+ textLabel.Size.Y.Offset
	)

	local absolutePosition = Vector2.new(
		textLabel.Position.X.Scale * viewportSize.X + textLabel.Position.X.Offset,
		textLabel.Position.Y.Scale * viewportSize.Y + textLabel.Position.Y.Offset
	)
	
	local screenSize, screenRatio, cornerCFrame = viewportToStuds(viewportSize, 1, absoluteSize)
	
	part.Size = Vector3.new(absoluteSize.X * screenRatio.X, absoluteSize.Y * screenRatio.Y, 0)
	part.CFrame = cornerCFrame * CFrame.new(absolutePosition.X * screenRatio.X, -absolutePosition.Y * screenRatio.Y, 0)
end)

7GrandDadPGN · February 10, 2024, 6:30am

Thank you! I’m dumbfounded behind the math on this one, but I can say its a very well thought answer!

RealmOfTheDeceased · September 2, 2024, 5:33pm

Hey, FrancklinDay! The function you made to convert gui space to world space is really helpful. A better way to calculate the offset position might be to subtract the parenting ScreenGui’s absolute position from the frame/text label’s:

local RunService = game:GetService("RunService")

local camera = game.Workspace.CurrentCamera
local part = game.Workspace.Part
local textLabel = script.Parent.Frame
local screenGui = textLabel:FindFirstAncestorOfClass("ScreenGui") -- example

local function getAbsolutes(Gui: ScreenGui, frame: GuiObject, viewportSize: Vector2)
	local absoluteSize = frame.AbsoluteSize
	local absolutePosition = frame.AbsolutePosition - Gui.AbsolutePosition
	
	return absoluteSize, absolutePosition
end

local function viewportToStuds(viewportSize, depth, absoluteSize)
	local aspectRatio = viewportSize.X / viewportSize.Y --Get screen aspect ratio (1920 / 1080)
	
	local heightFactor = math.tan(math.rad(camera.FieldOfView)/2) --Get height of screen from camera
	local widthFactor = aspectRatio * heightFactor --Use height to also get the width

	
	local screenSize = Vector2.new(-2 * widthFactor * -depth, -2 * heightFactor * -depth) --Get size of the screen
	local screenRatio = Vector2.new(screenSize.X / viewportSize.X, screenSize.Y / viewportSize.Y) --The ratio for pixel to studs
	local screenCFrame = camera.CFrame * CFrame.new(0, 0, -depth) --Get where the screen is in the world

	local cornerCFrame = 
		screenCFrame 
		* CFrame.new(-screenSize.X/2, screenSize.Y/2, 0) 
		* CFrame.new((absoluteSize.X * screenRatio.X)/2, -(absoluteSize.Y * screenRatio.Y)/2, 0)
	--Get the corner of the screen
	--part.CFrame = cornerCFrame
	
	return screenSize, screenRatio, cornerCFrame
end

RunService.RenderStepped:Connect(function()
	local viewportSize = camera.ViewportSize
	
	local absoluteSize, absolutePosition = getAbsolutes(screenGui, textLabel, viewportSize)
	
	local screenSize, screenRatio, cornerCFrame = viewportToStuds(viewportSize, 1, absoluteSize)
	
	part.Size = Vector3.new(absoluteSize.X * screenRatio.X, absoluteSize.Y * screenRatio.Y, 0)
	part.CFrame = cornerCFrame * CFrame.new(absolutePosition.X * screenRatio.X, -absolutePosition.Y * screenRatio.Y, 0)
end)

This method ensures that the frame’s absolute position is accurate to the screen inset no matter how many other frames it’s nested in.

Thanks again!