A point to be aware of is that pcall(0) still returns the same thread environment as outside the pcall. Also, I wouldn’t rely on pcall’s current functionality of making clearing the environment stack, that is more of an implementation detail which may change at some point (hopefully with warning if at all).
Since you’ve messaged me about my sandbox, I should mention that I’d be worried if you used it in production code since I have not exhaustively tried to break it. I must admit though, I do not know of a better solution.
A module runs in its own environment until it returns, only globals affect it. This means that all variables including anything accessed through game, script, or workspace will introduce variability to your script and is dangerous. Once it returns, the return value is another way to interact with the module. Any events connected to or callbacks given while the module was executing fall under global environment interactions. The return value must be properly secured as you described so that if a table is returned, it cannot be altered without the module’s permission. In addition, the script that required the module needs a way to verify the identity of the module. We will assume that only strings, numbers, booleans, userdatas, nil, and functions will exist in our return value. Outputted values need to be check so that they do not reveal information they shouldn’t. All are immutable and unchangeable from the outside except via userdata metamethods, function arguments, and the function’s environment.
If a module uses only local values then a changing environment will never affect it: values cannot be read or set that will have an effect. The only way to gain access to these variables in lua is through the debug library which Roblox has removed for security reasons. This allows us to shrink our examination to only global variables and function / metamethod arguments.
Arguments, of types string, number, and boolean are always safe and just need to be parameter checked. Interactions with userdatas and tables needs to be done very carefully since comparisons, indexing, and other interactions may cause a metamethod to be invoked and may cause unexpected behavior. Calling functions type, getmetatable, and rawequal as well as using userdatas and tables as keys in tables will not invoke any metamethods and can be used to work with them. Indexing them is dangerous and may result in infinite loops, errors, asynchronous operations, or unexpected behavior, exactly like calling a function argument. Return values from metamethods and functions should be treated with the same care as arguments. All arguments type should be identified and if it is a number, string, nil, or boolean then its value should be inspected to be within semantic program bounds.
Unless you specifically need need to work with them, I’d not accept arguments or return values of the thread type.
It should be noted that calling wait or any other asynchronous function (they are again only access through the global variables or arguments) may result in the script never resuming because it gives up the ability to run and may be permanently stopped by another script or the game. The state must be in a good point to quit when a calling any callback or a yielding lua / roblox function. Finally if your script runs too long without pausing or returning, expect to be rudely interrupted at an unknown location.
If the module always returns without running too long, localizes all state while loading, secures its return value, provides its identity, verifies callers identity where needed, only interacts with arguments without invoking any argument metamethods or functions along with only using synchronous lua calls, and performs value checking then negative interactions with other scripts / modules is not possible. Calls to metamethods or functions through globals and arguments can be done safely by expecting the function never to return (errors, infinite loops, async waits and thread removal), treating return values like arguments, and censoring call parameters.
The game engine / scripts with higher permissions may still be able to harm the module. I’m fairly confident that this post addresses all security issues relating to modules besides those dealing with semantic of the module’s interactions. Those, commonly known as “bugs” may be the most difficult threat to handle. I recommend creating unit and system tests along with adopting a functional programming style.