Help with snapping Objects to other Objects with different rotations | Scripting

I have this placement system with snapping to objects, and everything seems to work fine except with rotations.

When a object is rotated, let’s say 90 degrees on the Y, and then you try to snap it to another object that is rotated normally, 0 degrees on the y or just a different rotation, It doesn’t seem to snap right.
Here is whats happening:

image
As you can see, the snapping is not correct.
If I try it on that front surface or back for example, it goes slightly inside of it.

Here is the code where all of the calculations are taking place. It is inside of a RunService connection.

local function startPreview(obj: BasePart)
	if not obj or typeof(obj) ~= "Instance" then
		return false
	end
	
	-- connect a runservice function
	RunService:BindToRenderStep(properties.PREVIEW_RENDER_NAME, Enum.RenderPriority.Input.Value, function()
		-- get the mouses hit position
		local mouseRayVector = mouse.UnitRay
		-- define some raycast params
		local params = RaycastParams.new()
		params.FilterType = Enum.RaycastFilterType.Exclude
		params.FilterDescendantsInstances = {char, obj, FILTERING_FOLDER}

		local result = workspace:Raycast(mouseRayVector.Origin, mouseRayVector.Direction * 1000, params)

		local isValidPlacement
		local boxColor
		if result then
			-- set the surfacecolor3 of the selection box accordingly to distance
			local dist = (char.PrimaryPart.Position - obj.Position).Magnitude
			local hasOverlappingParts = checkOverlappingParts(obj)

			local inValidPlacement = dist >= properties.MAX_DISTANCE or hasOverlappingParts -- A variable that decides on whether a placement is invalid or valid

			isValidPlacement = not inValidPlacement

			boxColor = if inValidPlacement then properties.ERROR_COLOR_3 else properties.SUCCESS_COLOR_3
			
			-- get the results instance
			local instance = result.Instance :: BasePart
			local finalCF
			if instance:HasTag(player.UserId) then
				-- Snap to placed object with rotation taken into account
				local instanceCFrame = instance.CFrame
				local instanceSize = instance.Size
				local objSize = obj.Size

				-- Transform the normal into the local space of the snapped object
				local localNormal = instanceCFrame:VectorToObjectSpace(result.Normal)

				-- Calculate the snap offset
				local offset = (objSize / 2) + (instanceSize / 2)
				local snapOffset = Vector3.new(
					math.abs(localNormal.X) * offset.X,
					math.abs(localNormal.Y) * offset.Y,
					math.abs(localNormal.Z) * offset.Z
				)

				-- Transform the snap offset back to world space
				local worldOffset = instanceCFrame:VectorToWorldSpace(snapOffset)
				
				finalCF = instanceCFrame * obj.CFrame.Rotation + (result.Normal * worldOffset.Magnitude)
			else
				-- calculate the normal offset
				local normalOffset = result.Normal * (obj.Size * 0.5)
				
				finalCF = CFrame.new(result.Position + normalOffset) * obj.CFrame.Rotation
			end
			
			obj.CFrame = obj.CFrame:Lerp(finalCF, properties.LERP_SMOOTHING)
		else
			-- No result
			-- Mark it as invalid
			isValidPlacement = false

			-- set colors accordingly
			boxColor = properties.ERROR_COLOR_3

		end

		-- set the attribute of isvalidplacement
		obj:SetAttribute(properties.VALID_PLACEMENT_ATTRIBUTE, isValidPlacement)

		-- Set the color of the selection box now
		local selectionBox = obj:FindFirstChildWhichIsA("SelectionBox")

		selectionBox.Color3 = boxColor:Lerp(Color3.fromRGB(0, 0, 0), 0.25) -- a 'lerped' version
		selectionBox.SurfaceColor3 = boxColor -- normal version
	end)
	
	return true
end

It has to be something with the offset calculation, or the finalCF.

any help would be appreciated.