Given how long this has been an issue, I am surprised that I have only found one post mentioning it when searching for topics mentioning this issue(ViewportFrame rendering shouldn't have a resolution limit & should not have a white outline - #5 by joesmaller_xyz).
ViewportFrames look as though they have antialiasing on all channels except for the alpha channel, causing them to render a jarring outline around objects the same color as the BackgroundColor3 of the ViewportFrame object.
Notice the green glow in the above example; this gets particularly more noticeable for smaller ViewportFrames, and more complex models.
A common band-aid solution is to set the BackgroundColor3 to the same color as whatever is behind the ViewportFrame, but this only works if it’s a uniform and deterministic color:
If my assessment as to the cause of this issue is correct, it should be a very easy fix.