Weird collision underwater with player viewmodel

I made a custom viewmodel using the player’s arms (not cloned) but when swimming the collision is weird, like pushing me to random directions for no reason. Collision should be off for the viewmodel, so I have no idea what’s causing this
Here’s the viewmodel script below. Help appreciated!

local plr = game:GetService("Players").LocalPlayer
local char = plr.Character or plr.CharacterAdded:Wait()
local cam = workspace.CurrentCamera

local viewportModel = game:GetService("ReplicatedStorage").Viewport:Clone()
viewportModel.Parent = cam

local rShoulder = char:WaitForChild("Torso"):WaitForChild("Right Shoulder")

local rs = game:GetService("RunService")

char:WaitForChild("Right Arm").CanCollide = false
char:WaitForChild("Right Arm").Massless = true

plr.CameraMode = Enum.CameraMode.LockFirstPerson

rs.RenderStepped:Connect(function(dt)
	char:WaitForChild("Right Arm").LocalTransparencyModifier = char:WaitForChild("Right Arm").Transparency
	viewportModel:PivotTo((cam.CFrame * CFrame.new(0, -0.3, 0.3)) * CFrame.Angles(1, 0, 0)) --custom rot/pos values
	rShoulder.Part0 = viewportModel.Torso
end)

Just for a better look, I followed this tutorial for the viewmodel structure:
https://youtu.be/JKN7muCx1bA?si=h1B9CTBvkPWPWKdn

Bumping this just in case. Anyone got an idea?