Major Bug Regarding Map System

Hey everyone! I am attempting to make a minimap/map system, and am 90% complete with it, but have one major bug that is hindering my ability to completely finish everything.

The issue lies in the larger map (fullscreen version of the minimap). There are UIs above players/points of interests that line up with the viewport (the map), using the function camera:WorldToViewportPoint().

This all works fine, the issue is that there is parallax when scrolling on the map. If the UI/part that are tied to eachother are not in the direct center of the viewport, they are offset as if there is a z offset to the UI. This only happens in the X direction, the y direction is fine.

Notice how in the images below, when the bank is in the center of the viewport, the UI is not offset.


Here is all of the relevant code. I will also provide an image in case this is not as legibile.

Let me know if you have any further questions, thank you for your help!

local viewport = FullGui.Map.Viewport
			
			mapCamera.FieldOfView = mapSettings.Settings.FullMapFOV
			mapCamera.CFrame = CFrame.new(
				char.HumanoidRootPart.CFrame.Position.X, mapSettings.Settings.MapStartHeight + char.HumanoidRootPart.CFrame.Position.Y,
				char.HumanoidRootPart.CFrame.Position.Z) * CFrame.Angles(-math.pi/2, 0, 0)

			viewport.CurrentCamera = mapCamera
			
			while task.wait(mapSettings.Settings.RefreshTime) do -- Update full map
				
				-- START TEMP UI
				
				for _, v in pairs (viewport.Temp:GetChildren()) do v:Destroy() end -- reset temp positions
				
				for _, location: BasePart in pairs (ReplicatedStorage.MapDisplay.Locations:GetChildren()) do
					
					local locationLabel = ReplicatedStorage.UI.Main.Map.LocationTemplate:Clone(); locationLabel.Parent = viewport.Temp
					locationLabel.Header.Text = location.Name
					local screenPos: Vector3, onScreen: boolean = viewport.CurrentCamera:WorldToViewportPoint(location.Position)
					
					locationLabel.Position = UDim2.fromScale(screenPos.X, screenPos.Y)
					
					locationLabel.MouseButton1Down:Connect(function()
						ReplicatedStorage.Remotes.Teleportation:FireServer("location", location.Name)
					end)
					
				end
				
				for _, tempPlr: Player in pairs (game.Players:GetChildren()) do

					if tempPlr.Character then

						local tempChar = tempPlr.Character
						local HRP = tempChar:FindFirstChild("HumanoidRootPart")

						if not HRP then continue end

						HRP = HRP:Clone(); HRP.Parent = viewport.Temp
						
						-- FOR TESTING
						HRP.Transparency = 0
						HRP.Size = Vector3.new(25, 25, 25)
						-- END TESTING
						
						local screenPos: Vector3, onScreen: boolean = viewport.CurrentCamera:WorldToViewportPoint(HRP.Position)
						
						--if onScreen then

							local playerLabel = ReplicatedStorage.UI.Main.Map.MapPlayerLabel:Clone(); playerLabel.Parent = viewport.Temp
							playerLabel.Size = UDim2.fromScale(0.025, 0.025)
						
							if tempPlr.UserId == Player.UserId then
								playerLabel.PlayerText.Text = "YOU"
								playerLabel.BackgroundColor3 = Color3.fromRGB(0, 255, 0)
							else
								playerLabel.PlayerText.Text = tempPlr.Name
							end
						
						local fixedX = screenPos.X
						playerLabel.Position = UDim2.fromScale(fixedX, screenPos.Y)
						
						--end

					end

				end

				-- END TEMP UI
				
			end