Need help scaling special mesh

So, special meshes scale at double the scale of the part it is attached too, which results in out-of-place handle. My current fix makes it scale at half the ‘speed’, but this results in the initial click making the accessory half as small, then it continues to scale normally.

I really need help trying to figure out a way to keep it’s original size, but still scale it at the same speed, and I just cannot for the life of me figure out how.

This script is inside of a handle.

handles.MouseButton1Down:Connect(function()
	local dragC, mouseUpC

	local prevDist = 0 --The distance from last time handles.MouseDrag fired
	
	local bonusSpeed = 2
	
	specialMesh = handles.Adornee:FindFirstChildWhichIsA("SpecialMesh")
	
	dragC = handles.MouseDrag:Connect(function(face, distance)
		local deltaDist = (distance - prevDist)

		local resizeDir = Vector3.FromNormalId(face) -- converts it to a scalable range
		local finalResize = nil
		
		if resizeDir.X == -1 or resizeDir.Y == -1 or resizeDir.Z == -1 then -- check if input is negative
			finalResize = handles.Adornee.Size + (-1 * resizeDir * deltaDist)
		else
			finalResize = handles.Adornee.Size + (resizeDir * deltaDist)
		end
		
		-- Ensure the size stays within the min and max limits
		local clampedSize = Vector3.new(
			math.clamp(finalResize.X, MIN_SCALE.X, MAX_SCALE.X),
			math.clamp(finalResize.Y, MIN_SCALE.Y, MAX_SCALE.Y),
			math.clamp(finalResize.Z, MIN_SCALE.Z, MAX_SCALE.Z)
		)
		
		-- Check if the size actually changed
		local sizeChanged = (clampedSize ~= handles.Adornee.Size)
		handles.Adornee.Size = clampedSize
		
        -- PROBLEM CODE HERE
		if specialMesh then
			specialMesh.Scale = clampedSize / 2
		end
		
        -- Update position only if size changed
		if sizeChanged then
			local finalPos = handles.Adornee.Position + (handles.Adornee.CFrame:VectorToWorldSpace(resizeDir) * deltaDist / 2) -- update to worldspace so it scales along its actual axis
			handles.Adornee.Position = finalPos
		end

		prevDist = distance
	end)

	mouseUpC = handles.MouseButton1Up:Connect(function()
		dragC:Disconnect()
		mouseUpC:Disconnect()
	end)
end)

Current code demonstration:
(The scaling works properly, but it begins at half the size.)

Without dividing clamped size by 2:

Hierarchy:

If you are still confused, look at how the handle positions are occluded by the hat in the second video, then look at how the handle positions in the first video are not.

For anyone in the future, after a night of sleep, I found the best option was to double the size of the handle, which fixed the issue.