Complete semantic analysis (like type systems, constant propagation and folding, dead code elimination) are undecidable in languages with recursion and/or loops. The crux of the issue is that in those cases, data can manipulate code execution in infinite ways, causing slight differences in value that can propagate and make very large differences in program execution down the road. Most languages handle this by either requiring annotations from the programmer (like the explicit types in C, Java; programmers know more about the use cases than the compiler does), using runtime analysis of execution (JIT for scripting languages), or not analyzing the code. Optimizing or analyzing recursive functions and loops without additional information is undecidable and ultimately turns into a version of the halting problem.
Other features of Lua, mentioned by @ANSI_C have the same effect as the more general, above conditions. To make matters worse type inference is difficult in Lua because of operator overloading (val_1 + val_2 doesn’t tell you much about what the values are). The one advantage RBXLua has is that if you analyze an entire Lua program at once, it exists in a defined environment with types and expected values (userdata properties and methods).
I’ve thought a lot about Lua optimization and typing. The more I do, the more I want to write another language. But here is the kicker, even if you add type annotations to Lua or even design a whole new language, you still can only perform high level optimizations on Roblox. Eventually everything must be turned back into Lua source code to run, or be run by an interpreter written in Lua. Indeed, the future isn’t very bright for those of us interested in doing low level work like this on the Roblox platform.
As for outside Roblox optimization, I really prefer ASTs because they can be easily extended to hold additional information. The fundamental difference is pointers, you can edit one node without modifying the positions of the others or even create graphs which cannot be represented in a instruction based form without including “references” which would basically be pointers. ASTs are closer to what should actually exist in RAM while processing, while instruction based forms require interpretation before manipulation. But if you are optimizing outside Roblox, I’d reuse an existing format so that you can also reuse the optimization engine.
I mentioned what I think the future of Roblox Lua should be here: Get user’s script font size?
and again here Proper Way To Preload Assets where Saranok said, “Completely disagree. […] This is a core tenant of how we operate.” The future for work like this is very dim indeed, or at least it is to me.