Is doing this with pcall a code smell?

Since tab:method() is just sugar syntax for tab.method(tab), it theoretically means pcall(tab.method, tab) should work.

The question is, in your opinion is doing this over

   return tab:method()

a code smell?

It’s not really a code smell. It’s more of a preference based on what is more convenient versus what is more performant.

Let me name each pattern:

MethodCall:  Object:Method()
FuncCall:    Object.Method(Object)
FuncPCall:   pcall(Object.Method, Object)
MethodPCall: pcall(function() return Object:Method() end)

MethodPCall is usually used because it looks the most similar to MethodCall, or how the method would be called normally. Someone might turn to FuncPCall if they were looking to improve performance. We can run a benchmark to see how each pattern compares:

Function                Iterations  Nanoseconds/operation

BenchmarkMethodCall     41576583     29 ns/op
BenchmarkFuncCall       29480906     43 ns/op
BenchmarkFuncPCall      14762462     91 ns/op
BenchmarkMethodPCall    10577066    132 ns/op

MethodCalls are optimized by Luau, so they are always going to be faster then the equivalent FuncCall. PCalls require some extra setup, so they will always be somewhat slower than the equivalent non-PCall.

Going by this benchmark, the MethodCall optimizations are diminished by the overhead of pcall and wrapping the MethodCall in a function. On the other hand, MethodPCall is probably much more common than FuncPCall, so the pattern may end up getting optimized in the future.

Also remember that this applies only to the method call. If the method itself is expensive or is called infrequently, then the performance of the call is diminished by the performance of the method, so there is effectively no difference between any of the patterns.