Help with viewmodel wall detection

I recently tried creating the effect of the Viewmodel arms moving back to avoid clipping into the wall. It almost works fine but the arms sort of flicker when close to a wall. I know this is because the code uses the arm and gun muzzle’s position to calculate the arm length and distance from a wall and then modifies the arm’s position, which in turn changes the distance, causing a loop. Is there a way of doing this without causing the loop?

Code:

game:GetService("RunService"):BindToRenderStep("MoveArms", Enum.RenderPriority.Character.Value, function()
		--Creates Cframe which the Position of the VM arm's Position looking in the same direction as the muzzle
		local armPos = CFrame.lookAlong(vm.RightUpperArm.RightShoulderAttachment.WorldCFrame.Position, script.Parent.Handle.MuzzleExit.WorldCFrame.LookVector)
		--Creates Cframe which the local offset between the arm and muzzle
		local offset = armPos:ToObjectSpace(script.Parent.Handle.MuzzleExit.WorldCFrame)
		
		local params = RaycastParams.new()
		params.FilterType = Enum.RaycastFilterType.Exclude
		params.FilterDescendantsInstances = {vm, script.Parent.Parent}
		--casts a ray from the arm to get the distance from a wall
		local ray2 = workspace:Raycast((armPos * CFrame.new(offset.X, 0, 0)).Position, script.Parent.Handle.MuzzleExit.WorldCFrame.LookVector.Unit * 10, params)
		
		if ray2 then
			--Distance between the wall and gun muzzle
			local diff = ray2.Distance + offset.Z
			
			--if the difference is negative it means the muzzle is inside the wall
			---moves arms back the same distance to move arms out of wall
			if diff < 0 then
				print(diff)
				
				local rightCO = vm.RightUpperArm.RightShoulder
				local leftCO = vm.LeftUpperArm.LeftShoulder
				
				leftCO.C0 = CFrame.new(leftCO.C0.X, leftCO.C0.Y, 0.0019 - diff) * leftCO.C0.Rotation 
				rightCO.C0 = CFrame.new(rightCO.C0.X, rightCO.C0.Y, 0.0019 - diff) * rightCO.C0.Rotation 
			end
		end
	end)

Video: