Luau Recap: May 2020

Updates Announcements
#1

Luau (lowercase u, “l-wow”) is an umbrella initiative to improve our language stack - the syntax, compiler, virtual machine, builtin Lua libraries, type checker, linter (known as Script Analysis in Studio), and more related components. We continuously develop the language and runtime to improve performance, robustness and quality of life. Here we will talk about all things that happened since the update in March!

Many people work on these improvements; thanks @Apakovtac, @EthicalRobot, @fun_enthusiast, @zeuxcg!

If you have missed previous large announcements, here they are:

New function type annotation syntax

As noted in the previous update, the function type annotation syntax now uses : on function definitions and -> on standalone function types:

type FooFunction = (number, number) -> number

function foo(a: number, b: number): number
    return a + b
end

This was done to make our syntax more consistent with other modern languages, and is easier to read in type context compared to our old =>.

This change is now live; the old syntax is still accepted but it will start producing warnings at some point and will be removed eventually.

Number of locals in each function is now limited to 200

As detailed in Upcoming change to (correctly) limit the local count to 200 (which is now live), when we first shipped Luau we accidentally set the local limit to 255 instead of 200. This resulted in confusing error messages and code that was using close to 250 locals was very fragile as it could easily break due to minor codegen changes in our compiler.

This was fixed, and now we’re correctly applying limits of 200 locals, 200 upvalues and 255 registers (per function) - and emit proper error messages pointing to the right place in the code when either limit is exceeded.

This is technically a breaking change but scripts with >200 locals didn’t work in our old VM and we felt like we had to make this change to ensure long-term stability.

Require handling improvements in type checker + export type

We’re continuing to flesh out the type checker support for modules. As part of this, we overhauled the require path tracing - type checker is now much better at correctly recognizing (statically) which module you’re trying to require, including support for game:GetService.

Additionally, up until now we have been automatically exporting all type aliases declared in the module (via type X = Y); requiring the module via local Foo = require(path) made these types available under Foo. namespace.

This is different from the explicit handling of module entries, that must be added to the table returned from the ModuleScript. This was highlighted as a concern, and to fix this we’ve introduced export type syntax.

Now the only types that are available after require are types that are declared with export type X = Y. If you declare a type without exporting it, it’s available inside the module, but the type alias can’t be used outside of the module. That allows to cleanly separate the public API (types and functions exposed through the module interface) from implementation details (local functions etc.).

Improve type checker robustness

As we’re moving closer to enabling type checking for everyone to use (no ETA at the moment), we’re making sure that the type checker is as robust as possible.

This includes never crashing and always computing the type information in a reasonable time frame, even on obscure scripts like this one:

type ( ... ) ( ) ;
( ... ) ( - - ... ) ( - ... )
type = ( ... ) ;
( ... ) (  ) ( ...  )  ;
( ... ) ""

To that end we’ve implemented a few changes, most of them being live, that fix crashes and unbounded recursion/iteration issues. This work is ongoing, as we’re fixing issues we encounter in the testing process.

Better types for Lua and Roblox builtin APIs

In addition to improving the internals of the type checker, we’re still working on making sure that the builtin APIs have correct type information exposed to the type checker.

In the last few weeks we’ve done a major audit and overhaul of that type information. We used to have many builtin methods “stubbed” to have a very generic type like any or (...) -> any, and while we still have a few omissions we’re much closer to full type coverage.

One notable exception here is the coroutine. library which we didn’t get to fully covering, so the types for many of the functions there are imprecise.

If you find cases where builtin Roblox APIs have omitted or imprecise type information, please let us know by commenting on this thread or filing a bug report.

The full set of types we expose as of today is listed here for inquisitive minds: Roblox Luau type surface as of May 20, 2020 · GitHub

Removal of __gc from the VM

A bug with continue and local variables was reported to us a few weeks ago; the bug was initially believed to be benign but it was possible to turn this bug into a security vulnerability by getting access to __gc implementation for builtin Roblox objects. After fixing the bug itself (the turnaround time on the bug fix was about 20 hours from the bug report), we decided to make sure that future bugs like this don’t compromise the security of the VM by removing __gc.

__gc is a metamethod that Lua 5.1 supports on userdata, and future versions of Lua extend to all tables; it runs when the object is ready to be garbage collected, and the primary use of that is to let the userdata objects implemented in C to do memory cleanup. This mechanism has several problems:

  • __gc is invoked by the garbage collector without context of the original thread. Because of how our sandboxing works this means that this code runs at highest permission level, which is why __gc for newproxy-created userdata was disabled in Roblox a long time ago (10 years?)
  • __gc for builtin userdata objects puts the object into non-determinate state; due to how Lua handles __gc in weak keys (see Lua 5.2 Reference Manual), these objects can be observed by external code. This has caused crashes in some Roblox code in the past; we changed this behavior at some point last year.
  • Because __gc for builtin objects puts the object into non-determinate state, calling it on the same object again, or calling any other methods on the object can result in crashes or vulnerabilities where the attacker gains access to arbitrarily mutating the process memory from a Lua script. We normally don’t expose __gc because the metatables of builtin objects are locked but if it accidentally gets exposed the results are pretty catastrophic.
  • Because __gc can result in object resurrection (if a custom Lua method adds the object back to the reachable set), during garbage collection the collector has to traverse the set of userdatas twice - once, to run __gc and a second time to mark the survivors.

For all these reasons, we decided that the __gc mechanism just doesn’t pull its weight, and completely removed it from the VM - builtin userdata objects don’t use it for memory reclamation anymore, and naturally declaring __gc on custom userdata objects still does nothing.

Aside from making sure we’re protected against these kinds of vulnerabilities in the future, this makes garbage collection ~25% faster.

Memory and performance improvements

It’s probably not a surprise at this point but we’re never fully satisfied with the level of performance we get. From a language implementation point of view, any performance improvements we can make without changing the semantics are great, since they automatically result in Lua code running faster. To that end, here’s a few changes we’ve implemented recently:

  • A few string. methods, notably string.byte and string.char, were optimized to make it easier to write performant deserialization code. string.byte is now ~4x faster than before for small numbers of returned characters. For optimization to be effective, it’s important to call the function directly (string.byte(foo, 5)) instead of using method calls (foo:byte(5)). This had to be disabled due to a rare bug in some cases, this optimization will come back in a couple of weeks.
  • table.unpack was carefully tuned for a few common cases, making it ~15% faster; unpack and table.unpack now share implementations (and the function objects are equal to each other).
  • While we already had a very efficient parser, one long standing bottleneck in identifier parsing was fixed, making script compilation ~5% faster across the board, which can slightly benefit server startup times.
  • Some builtin APIs that use floating point numbers as arguments, such as various Vector3 constructors and operators, are now a tiny bit faster.
  • All string objects are now 8 bytes smaller on 64-bit platforms, which isn’t a huge deal but can save a few megabytes of Lua heap in some games.
  • Debug information is using a special compact format that results in ~3.2x smaller line tables, which ends up making function bytecode up to ~1.5x smaller overall. This can be important for games with a lot of scripts.
  • Garbage collector heap size accounting was cleaned up and made more accurate, which in some cases makes Lua heap ~10% smaller; the gains highly depend on the workload.

Library changes

The standard library doesn’t see a lot of changes at this point, but we did have a couple of small fixes here:

  • coroutine.wrap and coroutine.create now support C functions. This was the only API that treated Lua and C functions differently, and now it doesn’t.
  • require silently skipped errors in module scripts that occurred after the module scripts yielding at least once; this was a regression from earlier work on yieldable pcall and has been fixed.

As usual, if you have questions, comments, or any other feedback on these changes, feel free to share it in this thread or create separate posts for bug reports.

86 Likes
Luau Recap: August 2020
Luau Recap: June 2020
What is the '[unknown]' in 'custom' functions?
#5
#6

Please take the time to read the topic before replying! I’ll open it for replies in 30 minutes :slight_smile:

17 Likes
#7
#8

As part of this, we overhauled the require path tracing - type checker is now much better at correctly recognizing (statically) which module you’re trying to require, including support for game:GetService.

FINALLY!!! ah em, sorry about that… this is great; just to double check- it means that intellisense won’t behave weirdly with modules and GetService? If so, has this been released yet?

Type annotations are pretty cool. Any idea when they’ll be stable and released to production?

16 Likes
#9

This is awesome, is there any page on the devforum/wiki/github where we can track the API for Luau?
I want to convert my current game over to these optimizations but I’m having trouble trying to find some things.

1 Like
#11

I just noticed that newproxy no longer errors when called with an invalid argument, is this a bug? On the topic of newproxy, is there any reason that newproxy(proxy) was removed and could it come back?

I’ve also noticed a new function was added to the debug library, debug.loadmodule, which appears to load a module and return a the function, is its purpose to circumvent caching with require? It also appears that you can pass arguments to the module and return 0 values or more than 1 value, which is useful, I guess?

Module:

print(...)
return select("#",...)+1,select("#",...)+2,select("#",...)+3

Script:

local d = debug.loadmodule(workspace.ModuleScript)
print(d(1,2,3))
print(d(1,2,3,4))
--> 1 2 3
--> 4 5 6
--> 1 2 3 4
--> 5 6 7
4 Likes
#12

Yeah, it looks like the newproxy issue is a bug. For whatever reason, as long as a non-false argument is provided to newproxy, a userdata object is created with a brand new metatable, which causes newproxy(proxy) to not work as intended. Although I’m curious as to what new update could’ve caused this, because, from a quick skim, nothing related to this seems to be mentioned here or in the recent release notes thread.

1 Like
#13

I believe this is because cloning builtin userdata objects with newproxy is unsafe.

The code for newproxy was cleaned up after removing __gc and it looks like we probably need that error check back. Should be fixed next week hopefully.

1 Like
#14

Awesome work! I’m confused about one thing though:

Previously, would calling Coroutine.wrap() on C API’s like the marketplace service error (I never tried it)? Or does this just mean that internally it follows a similar path for C code and lua code when making a new thread?

#15

https://www.lua.org/source/5.1/lbaselib.c.html#luaB_newproxy

static int luaB_newproxy (lua_State *L) {
  lua_settop(L, 1);
  lua_newuserdata(L, 0);  /* create proxy */
  if (lua_toboolean(L, 1) == 0)
    return 1;  /* no metatable */
  else if (lua_isboolean(L, 1)) {
    lua_newtable(L);  /* create a new metatable `m' ... */
    lua_pushvalue(L, -1);  /* ... and mark `m' as a valid metatable */
    lua_pushboolean(L, 1);
    lua_rawset(L, lua_upvalueindex(1));  /* weaktable[m] = true */
  }
  else {
    int validproxy = 0;  /* to check if weaktable[metatable(u)] == true */
    if (lua_getmetatable(L, 1)) {
      lua_rawget(L, lua_upvalueindex(1));
      validproxy = lua_toboolean(L, -1);
      lua_pop(L, 1);  /* remove value */
    }
    luaL_argcheck(L, validproxy, 1, "boolean or proxy expected");
    lua_getmetatable(L, 1);  /* metatable is valid; get it */
  }
  lua_setmetatable(L, 2);
  return 1;
}

https://www.lua.org/source/5.1/lbaselib.c.html#base_open

  /* `newproxy' needs a weaktable as upvalue */
  lua_createtable(L, 0, 1);  /* new table `w' */
  lua_pushvalue(L, -1);  /* `w' will be its own metatable */
  lua_setmetatable(L, -2);
  lua_pushliteral(L, "kv");
  lua_setfield(L, -2, "__mode");  /* metatable(w).__mode = "kv" */
  lua_pushcclosure(L, luaB_newproxy, 1);
  lua_setglobal(L, "newproxy");  /* set global `newproxy' */

newproxy puts the metatable into a weak table stored as an upvalue when it creates an object on newproxy(true), which is then checked when newproxy(proxy) to ensure that newproxy(proxy) can’t clone any value. Since newproxy(proxy) verifies that the metatable of the object came from an object created by newproxy, there shouldn’t be any issues?

#16

Hmm, you may be right. I think the original disablement was a sandboxing measure, but it’s unfortunately impossible to figure out why it was done now. Regardless newproxy actually gets in the way of some VM work so I don’t have any motivation to re-enable this :smiley:

#17

Previously, coroutine.create(coroutine.yield) would error with “Lua function expected”, since the only function you could pass to coroutine.create was a Lua function; now* you can pass either a C function or a Lua function, as is usually the case with other APIs.

* once the change gets enabled, so in an hour or so.

2 Likes
#18

This is certainly getting interesting.

@zeuxcg, can you possibly convince me (and potentially other developers) to review thousands of lines of code to use typechecking?

#19

I am once again asking you for typing support for varargs. It’s important for Lua and even built-in functions (like bit32.band) use it. Also, is there a timeline on a replacement for as?

Those two things are the main things preventing me from using typed Lua in its current state.

Other than that, glad that string.byte and string.char are getting optimizations. Can I get some clarification on what small number of returned characters means though? How many characters is too many?

I also appreciate the type information for built-in API. It’s a great example of how to use the type system in certain situations and satisfies my “knowledge for its own sake” craving. Would love a way to generate the information ourselves but I understand that may never happen.

Also, would it be possible to get a list of what in all has specific optimizations in the future? People shouldn’t have to trawl through months of recaps to learn what is and isn’t optimized. While in a perfect world it wouldn’t matter to people, it has to be somewhere as a reference – as an example, a lot of online resources say that ipairs is slow and shouldn’t be used (and in vanilla Lua it shouldn’t be) but in Luau it’s blazingly fast and should be used. That sort of information is important.

4 Likes
#20

No word on either yet; for as we have yet to find a good backwards compatible syntax, and better typing for varargs is a bit involved. Out of curiosity what do you need as for that you can’t accomplish without it?

This is just a reflection of the fact that getting, say, 100 characters using string.byte may not see significant improvements from this change. For the purpose of this note “small” is “1-10”.

It actually will, but timeline is unclear.

This is a good idea but we don’t have a great way to host and update this information right now. I’d say in general we’re trying to make it so that you don’t need to remember arbitrary rules. For example, iteration with ipairs/pairs is completely idiomatic; advice to avoid ipairs mostly stems from it causing trace aborts in LuaJIT; for us the guidance is basically “use the most common way to achieve the result and talk to us if it’s slow”.

1 Like
#21

I ran into a case with my current project where string.byte returning number|nil was problematic and it’s not elegant to fix it.

The code snippet in question is basically this:

local source = "foo"
local bytes = table.create(#source, 0)
for i = 1, #source do
  bytes[i] = string.byte(source, i)
end

As you would expect, this causes a type mismatch between number and number|nil. Fixing it at the moment is possible by manually checking if it exists, but that’s a bad solution and makes the code look messy:

local source = "foo"
local bytes = table.create(#source, 0)
for i = 1, #source do
  local byte = string.byte(source, i)
  if type(byte) ~= "nil" then
    bytes[i] = byte
  end
end

So it’s not really a case of not being able to accomplish something so much as it makes me jump through hoops to make the type system infer something I already know is true, which is annoying and makes me not want to use the type system.

A possible solution for this case at least might be to add ! as a type assertion operator like TypeScript does. That would cover most of the issues I can think of.

As for as in general… That is a bit of a conundrum. There are a couple ‘obvious’ ways to implement that, ranging from C’s style to the alternate style that TypeScript offers, but none of them are very good and come with their own parser problems. Is the number of games using as really substantial enough that it’s worth worrying about making a breaking change? I know there were some issues when it was initially turned on, but I didn’t think it was that many.

1 Like
#22

As for the as syntax, there was a discussion elsewhere and an interesting workaround came to mind. Maybe instead of adding new syntax, we could instead get a new global function or such that simply takes any as its arguments and returns any? Since it seems the type checker doesn’t complain when you erase the type like this, it could work and be backwards compatible.
image

It’s already possible and seems to work, but it could always be built in as a general solution. The value can just be bound to a typed variable or used in a type context like above to give it a type.

1 Like
#23

Will Luau replace Lua later? if so then you should probably make it optional

#24

It already has. Also why would it be optional? There’s no reason to use the old Lua anymore. If you’re worried about the new type syntax, that’s optional and you don’t need to write code with it.

2 Likes