Simple VR grabbing system

I’m trying to make a simple VR grabbing system with Nexus VR. This is mostly for steering wheels, hand brakes, levers, car doors, etc, stuff that has hinges. But I would also like to be able to move some objects. The problem is, none of these work.

Whenever I try to grab something, the part I am grabbing gets welded to my hand but it’s floating really far from it. I could grab it with my hand initially inside the part, yet after grabbing it, it would be on top of my head. I don’t know why this is happening and how to fix it.

I have tried searching through the developer forum but I haven’t found anything that fixed it. I don’t wanna use AlignPosition or AlignOrientation because I’m pretty sure it wouldn’t work for the things I want it the grabbing system to be used for.

Client script
local VRService = game:GetService("VRService")
local UIS = game:GetService("UserInputService")
local Players = game:GetService("Players")
local ReplicatedStorage = game:GetService("ReplicatedStorage")

if not VRService.VREnabled then script:Destroy() end

local VRHandler = ReplicatedStorage.Events.ClientToServer:WaitForChild("VRHandler")

local Player = Players.LocalPlayer
local Character = Player.Character or Player.CharacterAdded:Wait()

script.Parent = Character

local RightHand = Character.RightHand
local LeftHand = Character.LeftHand

local gripHeld = {
	Left = false,
	Right = false,
}

function Grab(hand: BasePart): ()
	for _, part in pairs(workspace:GetPartsInPart(hand)) do
		if part:HasTag("VRGrab") then
			print(part.Name)
			if hand == LeftHand then
				gripHeld.Left = true
			elseif hand == RightHand then
				gripHeld.Right = true
			end
			
			local weld = hand:FindFirstChild("VRWeld")
			if weld then
				local table = {
					Action = "Weld",
					Actor = weld,
					Target = part
				}
				VRHandler:FireServer(table)
			else
				warn("No weld found for hand "..hand.Name)
			end
			
			break
		end
	end
end

function Ungrab(hand: BasePart): ()
	local weld = hand:FindFirstChild("VRWeld")
	if weld then
		VRHandler:FireServer({
			Action = "Unweld",
			Actor = weld
		})
	else
		warn("No weld found for hand "..hand.Name)
	end
	
	if hand == LeftHand then
		gripHeld.Left = false
	elseif hand == RightHand then
		gripHeld.Right = false
	end
end

UIS.InputBegan:Connect(
	function(inp, proc)
		if proc then return end
		if inp.KeyCode == Enum.KeyCode.ButtonR1 then
			print("ButtonR1 grab")
			Grab(RightHand)
		elseif inp.KeyCode == Enum.KeyCode.ButtonL1 then
			print("ButtonL1 grab")
			Grab(LeftHand)
		end
	end
)
UIS.InputEnded:Connect(
	function(inp, proc)
		if proc then return end
		if inp.KeyCode == Enum.KeyCode.ButtonR1 then
			print("ButtonR1 ungrab")
			Ungrab(RightHand)
		elseif inp.KeyCode == Enum.KeyCode.ButtonL1 then
			print("ButtonL1 ungrab")
			Ungrab(LeftHand)
		end
	end
)
Server script
local ReplicatedStorage = game:GetService("ReplicatedStorage")

local VRHandler = ReplicatedStorage.Events.ClientToServer:WaitForChild("VRHandler")

VRHandler.OnServerEvent:Connect(function(plr, info)
	local Action = info.Action
	local Actor = info.Actor
	local Target = info.Target
	
	if Action then
		if Action:lower() == "weld" then
			if (Actor and Actor:IsA("WeldConstraint")) and (Target and Target:IsA("BasePart")) then
				Actor.Part1 = Target
				Target.Anchored = false
				Target:SetNetworkOwner(plr)
			else
				warn("No actor or target found; "..(Actor.Name or "nil, ")..(Target.Name or "nil"))
			end
		elseif Action:lower() == "unweld" then
			if (Actor and Actor:IsA("WeldConstraint")) then
				Actor.Part1:SetNetworkOwner(nil)
				Actor.Part1 = nil
			else
				warn("No actor found; "..(Actor.Name or "nil"))
			end
		end
	else
		warn("No action received")
	end
end)

I have no idea how to go about this. If someone already has a system that does this or something similar, I would appreciate it if you share how you did it.
Thanks in advance

You need to change the position of the item being grabbed to be the hand, and then weld it. Modified your code sample here:

if Action then
		if Action:lower() == "weld" then
			if (Actor and Actor:IsA("WeldConstraint")) and (Target and Target:IsA("BasePart")) then
				Target.CFrame = info.TargetCFrame -- assuming you send the position to weld it at as well
				Actor.Part1 = Target
				Target.Anchored = false
				Target:SetNetworkOwner(plr)
			else
				warn("No actor or target found; "..(Actor.Name or "nil, ")..(Target.Name or "nil"))
			end

I already had this earlier but it didn’t work either, and I don’t think it would be very friendly with parts that have hinges and other constraints. Obviously I will have to separate items from other stuff, so I could change it, but right now I just want the welding to work.

Well, what was the issue that you had before when trying to move parts before welding?

The part wouldn’t move to the hand

Did you move it on the client or the server, before or after setting the network ownership?

The client tells the server to “weld it”, and it gets welded after setting the network ownership to the client

The client has network ownership of the object. You need to either weld it on the client after it has ownership of it, or weld it on the server before the client has ownership.