How can I make my custom camera collision smoother?

There’s no other alternative solution to your problem.

You could do something like:

Camera_Distance = math.min(rayDistance,
     math.lerp(Camera_Distance,Expected_Camera_Distance,Delta_Time*10))