Correct me if I am wrong, but I believe your math.sign() functions are only limited to one axis, so when the boat is facing in the Z direction, your math.sign() for angleFront is still measuring the difference in X between the portPos and bowPos, which in that instance would be equal. Same goes with angleSide. It doesn’t seem like the angles of the boat are going too crazy so signage might be the issue as you suspected. Let me know if this is true
On second thought, the differences in height in hFront and hSide should yield you a sign already, so is it actually necessary to have math.sign() functions to apply signage to your angle?
local boatPos = -- Your boat's position, or 0,0,0 if this CFrame is only going to be used for rotation
local xVector = (portPos - starboardPos).Unit
local zVector = (bowPos - sternPos).Unit
local yVector = zVector:Cross(xVector)
local targetCFrame = CFrame.fromMatrix(boatPos, xVector, yVector, zVector)