Why are metamethods unable to be yielded?

I’m currently working on a project where scripts are able to index a table to get data from another script. For example:

Script 1:

local Script2 = setmetatable({},{
  __index = function(Self, Index)
    local ReturnEvent = Instance.new("BindableEvent")
    local Return
            
    ReturnEvent.Event:Once(function(...)
        Return = {...}
    end)
            
    InvokeMethod:Fire(ReturnEvent, "__index", Index)
    
    repeat
        
    until Return ~= nil
    
    return table.unpack(Return)
  end
})

Script 2:

local OnEvent = function(ReturnEvent: BindableEvent?, Metamethod: string, ...)
    if ModuleData[Metamethod] then
        local Return = ModuleData[Metamethod](ModuleData, ...)
        
        if ReturnEvent then ReturnEvent:Fire(Return) end
    end
end

Invoke.Event:Connect(OnEvent)

In development, everything appears in a typical modular heirarchy, in the style of modules being loaded in to a central table, and all submodules index that table to get data from other modules. The problem with this approach is that the script that require()s all the modules will be handling every function across every module, and having an entire game running in one Luau VM is obviously a bad idea.

My approach is to split every module into its own server script, which Script 2 above contains code for the template script which is :Clone()d whenever a module is loaded, and to get data like that. However, one of the biggest issues with this approach is being unable to yield (task.wait, wait, delay, :Wait() members of events) the __index function, erroring with Attempt to yield across metamethod / C-call boundry. Is there a particular reason as to why metamethods cannot be yielded?

The metamethods are probably meant to be atomic, it would not be safe to yield from inside them. This is just speculation about Lua’s design, though.

int lua_yield(lua_State* L, int nresults)
{
    if (L->nCcalls > L->baseCcalls)
        luaG_runerror(L, "attempt to yield across metamethod/C-call boundary");
    L->base = L->top - nresults; // protect stack slots below
    L->status = LUA_YIELD;
    return -1;
}

This is from the luau source code (src/ldo.cpp), and frankly I have no idea what this means but this is what’s causing it to error. Unsure why.

struct lua_State for reference:

struct lua_State {
  CommonHeader;
  lu_byte status;
  StkId top;  /* first free slot in the stack */
  StkId base;  /* base of current function */
  global_State *l_G;
  CallInfo *ci;  /* call info for current function */
  const Instruction *savedpc;  /* `savedpc' of current function */
  StkId stack_last;  /* last free slot in the stack */
  StkId stack;  /* stack base */
  CallInfo *end_ci;  /* points after end of ci array*/
  CallInfo *base_ci;  /* array of CallInfo's */
  int stacksize;
  int size_ci;  /* size of array `base_ci' */
  unsigned short nCcalls;  /* number of nested C calls */
  unsigned short baseCcalls;  /* nested C calls when resuming coroutine */
  lu_byte hookmask;
  lu_byte allowhook;
  int basehookcount;
  int hookcount;
  lua_Hook hook;
  TValue l_gt;  /* table of globals */
  TValue env;  /* temporary place for environments */
  GCObject *openupval;  /* list of open upvalues in this stack */
  GCObject *gclist;
  struct lua_longjmp *errorJmp;  /* current error recover point */
  ptrdiff_t errfunc;  /* current error handling function (stack index) */
};

It looks like the check is designed to prevent incorrect ordering of the call stack. This is a design choice to limit the complexity of C hooks. It is stopping you from yielding, calling a metamethod which calls a C function, then performing a Lua yield again. For whatever reason this isn’t allowed.