Help changing orientation of part/model based on mouse movement

So, I am trying to achieve object rotation based on mouse movement. As the mouse moves, orientation of the object changes. But the problem is that the orientation seems to be based on the front only. If I go to another side of the part, the orientation is based on the front.

Some solutions that I have tried involved using UserInputService:GetMouseDelta() but this requires your mouse to be locked or in first person. Here is a visual of this problem:

This is how I want it to function:

This is my current code:

local PartToRotate = workspace:WaitForChild("Part")
local UserInputService = game:GetService("UserInputService")
local TweenService = game:GetService("TweenService")
local Camera = workspace.CurrentCamera
local Mouse = game:GetService("Players").LocalPlayer:GetMouse()
local start = Vector2.new(Mouse.X, Mouse.Y)
local down = false

UserInputService.InputBegan:Connect(function(inputObject)
    if inputObject.UserInputType == Enum.UserInputType.MouseButton1 then
        start = Vector2.new(Mouse.X, Mouse.Y)
        down = true
    end
end)

UserInputService.InputEnded:Connect(function(inputObject)
    if inputObject.UserInputType == Enum.UserInputType.MouseButton1 then
        down = false
    end
end)

game:GetService("RunService").RenderStepped:Connect(function()
    if down then
        local currentPosition = Vector2.new(Mouse.X, Mouse.Y)
        local delta = (start - currentPosition) * 1  -- Adjust this multiplier to control the rotation speed

        if delta.X == 0 and delta.Y == 0 then
            return
        end

        local newOrientation = Vector3.new(PartToRotate.Orientation.X, PartToRotate.Orientation.Y + (-delta.X), PartToRotate.Orientation.Z - delta.Y)

        local tweenInfo = TweenInfo.new(0.1, Enum.EasingStyle.Linear)--adjust style and time to whatever you want
        local goal = {Orientation = newOrientation}
        local tween = TweenService:Create(PartToRotate, tweenInfo, goal)
    
        tween:Play()
        start = currentPosition 
    end
end)

I was thinking this problem could be fixed maybe by involving the Camera’s LookVector, but Im not fully sure how or why.

bump… (added chars to reach limit)