Luau Recap: August 2020

[zeuxcg]

As everyone knows by now, Luau is our new language stack that you can read more about at Luau - Luau and the month following June is August so let’s talk about changes, big and small, that happened since June!

Many people work on these improvements, with the team slowly growing - thanks @Apakovtac, @EthicalRobot, @fun_enthusiast, @mrow_pizza and @zeuxcg!

If you have missed previous large announcements, here they are:

• Faster Lua VM Released
• Luau Recap: November 2019
• Luau Type Checking Beta!
• Luau Recap: February 2020
• Luau Recap: March 2020
• Luau Recap: May 2020
• Luau Recap: June 2020

Type annotations are safe to use in production!

When we started the Luau type checking beta, we’ve had a big warning sign in the post saying to not publish the type-annotated scripts to your production games which some of you did anyway. This was because we didn’t want to commit to specific syntax for types, and were afraid that changing the syntax would break your games.

This restriction is lifted now. All scripts with type annotations that parse & execute will continue to parse & execute forever. Crucially, for this to be true you must not be using old fat arrow syntax for functions, which we warned you about for about a month now:

… and must not be using the __meta property which no longer holds special meaning and we now warn you about that:

Part of the syntax finalization also involved changing the precedence on some type annotations and adding support for parentheses; notably, you can now mix unions and intersections if you know what that means ((A & B) | C is valid type syntax). Some complex type annotations changed their structure because of this - previously (number) -> string & (string) -> string was a correct way to declare an intersection of two function types, but now to keep it parsing the same way you need to put each function type in parentheses: ((number) -> string) & ((string) -> string).

Type checking is not out of beta yet - we still have some work to do on the type checker itself. The items on our list before going out of beta right now include:

• Better type checking for unary/binary operators
• Improving error messages to make type errors more clear
• Fixing a few remaining crashes for complex scripts
• Fixing conflation of warnings/errors between different scripts with the same path in the tree
• Improving type checking of globals in nonstrict mode (strict mode will continue to frown upon globals)

Of course this doesn’t mark the end of work on the feature - after type checking goes out of beta we plan to continue working on both syntax and semantics, but that list currently represents the work we believe we have left to do in the first phase - please let us know if there are other significant issues you are seeing with beta beyond future feature requests!

The work on parallel Luau has begun

We’ve kept you in the dark for a while about this; this is because we were working on a full specification for how multithreading will work in Luau. This work was finished in June and we’ve started working on an implementation.

Our plans here are grand and expansive, and this work will ship in phases, with each successive phase allowing more and more engine functionality to be accessible from parallel context. The details will be shared when we’re closer to a beta release, which should happen by the end of the year.

Improved type checking for Roblox builtin types

Previously we used to model Roblox userdata types as tables with a shape dictated by the available members. This worked fine most of the time but resulted in a few odd cases where the type checker thought some types were compatible which wasn’t true in runtime; for example:

• Folder was equivalent to Instance because Folder class didn’t introduce new members
• You could pass a CFrame to a function that is declared to accept a Vector3 because CFrame had X/Y/Z members

To fix that we introduced special support for Roblox builtin types called “nominal types”, where some types are defined by their name, not by their shape, and have a subtyping relationship with other types (e.g. Folder is-a Instance). Right now this is limited to Roblox userdata types.

We also fixed some cases in Roblox API where an Instance type accidentally used any during type checking.

Format string analysis

A few standard functions in Luau are using format strings to dictate the behavior of the code. There’s string.format for building strings, string.gmatch for pattern matching, string.gsub's replacement string, string.pack binary format specification and os.date date formatting.

In all of these cases, it’s important to get the format strings right - typos in the format string can result in unpredictable behavior at runtime including errors. To help with that, we now have a new lint rule that parses the format strings and validates them according to the expected format.

Right now this support is limited to direct library calls (string.format("%.2f", ...) and literal strings used in these calls - we may lift some of these limitations later to include e.g. support for constant locals.

Additionally, if you have type checking beta enabled, string.format will now validate the argument types according to the format string to help you get your %ds and %ses right.

Improvements to string. library

We’ve upgraded the Luau string library to follow Lua 5.3 implementation; specifically:

• string.pack/string.packsize/string.unpack are available for your byte packing needs
• string.gmatch and other pattern matching functions now support %g and \0 in patterns

This change also [inadvertently] makes string.gsub validation rules for replacement string stricter - previously % followed by a non-digit character was silently accepted in a replacement string, but now it generates an error. This accidentally broke our own localization script (Purchase Prompt broken in some games (% character in title) - #8 by OutlookG), but we got no other reports, and this in retrospect is a good change as it makes future extensions to string replacement safe… It was impossible for us to roll the change back and due to a long release window because of an internal company holiday we decided to keep the change as is, although we’ll try to be more careful in the future.

On a happier note, string.pack may seem daunting but is pretty easy to use to pack binary data to reduce your network traffic (note that binary strings aren’t safe to use in DataStores currently); I’ve posted an example in the release notes thread (Release Notes for 441 - #35 by zeuxcg) that allows you to pack a simple character state in 16 bytes like this:

local characterStateFormat = "fffbbbB"

local characterState = string.pack(characterStateFormat,
    posx, posy, posz, dirx * 127, diry * 127, dirz * 127, health)

And unpack it like this after network transmission:

local posx, posy, posz, dirx, diry, dirz, health =
    string.unpack(characterStateFormat, characterState)
dirx /= 127
diry /= 127
dirz /= 127

Assorted fixes

As usual we fixed a few small problems discovered through testing. We now have an automated process that generates random Luau code in semi-intelligent ways to try to break different parts of our system, and a few fixes this time are a direct result of that.

• Fix line debug information for multi-line function calls to make sure errors for code like foo.Bar(...) are generated in the appropriate location when foo is nil
• Fix debug information for constant upvalues; this fixes some bugs with watching local variables from the nested functions during debugging
• Fix an off-by-one range check in string.find for init argument that could result in reading uninitialized memory
• Fix type confusion for table.move target table argument that could result in reading or writing arbitrary memory
• Fix type confusion for debug.getinfo in some circumstances (we don’t currently expose getinfo but have plans to do so in the future)
• Improve out of memory behavior for large string allocations in string.rep and some other functions like table.concat to handle these conditions more gracefully
• Fix a regression with os.time from last update, where it erroneously reverted to Lua 5.x behavior of treating the time as a local time. Luau version (intentionally) deviates from this by treating the input table as UTC, which matches os.time() behavior with no arguments.

Performance improvements

Only two changes in this category here this time around; some larger scale performance / memory improvements are still pending implementation.

• Constant locals are now completely eliminated in cases when debugging is not available (so on server/client), making some scripts ~1-2% faster
• Make script compilation ~5% faster by tuning the compiler analysis and code generation more carefully

Oh, also math.round is now a thing which didn’t fit into any category above.

[RafDev]

Great work, you’re visibly making Roblox a better platform to develop on! Seriously, this project always hypes me up.

---

@zeuxcg

Oh, also math.round is now a thing

I felt a great disturbance in the Force, as if millions of voices suddenly pressed “replace all” on a math.floor(x+0.5) find prompt.

[DrKittyWaffles (x64dbg)]

How does math.round work. How do we specify if we want to round down or up.

[zeuxcg]

math.round always rounds to the nearest integer; math.floor rounds down and math.ceil rounds up.

[nicemike40]

@zeuxcg

• Constant locals are now completely eliminated in cases when debugging is not available (so on server/client), making some scripts ~1-2% faster
• Make script compilation ~5% faster by tuning the compiler analysis and code generation more carefully

Would love a blog post on zeux.io about the deets of the static analysis for Luau, if that was your project . I know you’re as excited about the new stack as we are, thank you for all of this amazing work!

[DrKittyWaffles (x64dbg)]

What if my input is 0.5. Does it round towards 0 or 1. Conventionally I would round towards 1, but the neighboring integers are equidistant.

[zeuxcg]

The tie-breaking rules match that of C99 and break the tie by rounding halfway numbers away from zero.

[Dekkonot]

math.round is very cool, thank you.

With the removal of __meta, is there no way for us to define the result of various operators now? This feels like it would be useful.

Also if we’re approaching the end of the typed Lua beta, does that mean I can start bothering you guys about vararg support again?

[zeuxcg]

@Dekkonot

With the removal of __meta , is there no way for us to define the result of various operands now? This feels like it would be useful.

Right now the “best” way is to use typeof(setmetatable(...)). This is not perfect but in general right now the best way to specify a type for an object seems to be via typeof. We plan to explore some way to define classes or interfaces or both to fix this in the future, but we decided to start by removing the __meta hack.

@Dekkonot

Also if we’re approaching the end of the typed Lua beta, does that mean I can start bothering you guys about vararg support again?

Yeah please share your grievances with varargs here, we’d love to make this better.

[DrKittyWaffles (x64dbg)]

Last question I promise.

Is there absolutely any difference at the end if I do math.floor(x + .5) rather than math.round(x)

I decided to run a quick test (probably should’ve done this for my previous question too ) and I found that math.round(x) is about 3 times slower than math.floor(x + .5). I was hoping it would be faster

do
	local start = tick()
	
	for i = 1, 10, .0001 do
		math.round(i)
	end
	
	print(tick() - start)
end

do
	local start = tick()
	
	for i = 1, 10, .0001 do
		math.floor(i + .5)
	end
	
	print(tick() - start)
end

  0.0032095909118652
  0.0009310245513916
> print(0.0032095909118652/0.0009310245513916)
3.4473751600512

[zeuxcg]

math.floor has some magical optimizations that haven’t been implemented for math.round yet. There’s some difference in behavior, math.round in particular returns 0 correctly for, uh, 0.49999999999999994.

[Blockzez]

Is it possible to change the math.round function to the nearest even?
I’m used to half even rounding and it just feels natural to me.

[tnavarts]

@RafDev

I felt a great disturbance in the Force, as if millions of voices suddenly pressed “replace all” on a math.floor(x+0.5) find prompt.

WARNING: Do not use round for geometric snapping!!, you should still be using math.floor(x + 0.5) for that.

math.round() makes sense for rounding textual inputs from the user, but is not the correct thing to use for geometric snapping, such as snapping a dragged part or Gui to the nearest grid unit, thanks to the difference in rounding rules:

Good:

math.floor(-1.5 + 0.5) -> -1
math.floor(-0.5 + 0.5) -> 0
math.floor(0.5 + 0.5) -> 1
math.floor(1.5 + 0.5) -> 2

Bad:

math.round(-1.5) -> -2
math.round(-0.5) -> -1
math.round(0.5) -> 1  ... +1 to the input resulted in +2 to the output, yikes!
math.round(1.5) -> 2

TL;DR: The fact that math.floor / math.ceil always have the same relative “bias” in their rounding is desirable for geometric snapping.

[DataBrain (Amber)]

@zeuxcg

… and must not be using the __meta property which no longer holds special meaning and we now warn you about that:

Will we ever see syntax for defining metatable types? A built-in type (something like WithMetatable<T, U>) like the ones TypeScript has (i.e. FunctionArguments, ReturnType, Partial) would be helpful.

I guess this is technically possible right now?

type WithMetatable<T, U> = typeof((function()
	local x: T
	local y: U
	return setmetatable(x, y)
end)())

Luau likes to crash when I mess around with this too much though

edit: This code will 100% of the time crash luau when pasted into a script:

--!strict
type MyType<T, U> = typeof((function(x, y)
	
end))

local x: MyType<{}, {}> = {}

[colbert2677 (ImagineerColbert)]

I’m excited to know that parallel Luau is under way. I’m not too knowledgeable on the technical implications of dispatching threads to handle tasks or how it’ll all go down, but I’m keeping posted. If this will allow running of certain expensive tasks in different threads and if this can increase performance depending on how it’s used, then it’s absolutely something I’ll want to have access to.

[MFCmaster1234]

Am I missing something regarding the “Month following June is August” bit? It feels like I am missing some context regarding that statement. Otherwise, I am really excited about this.

[Halalaluyafail3 (FindFirstChild)]

@zeuxcg

string.gmatch and other pattern matching functions now support %g and \0 in patterns

Since \0 is allowed in patterns, will utf8.charpattern be changed to use \0 instead of %z?

@zeuxcg

Format string analysis

A few standard functions in Luau are using format strings to dictate the behavior of the code. There’s string.format for building strings, string.gmatch for pattern matching, string.gsub 's replacement string, string.pack binary format specification and os.date date formatting.

In all of these cases, it’s important to get the format strings right - typos in the format string can result in unpredictable behavior at runtime including errors. To help with that, we now have a new lint rule that parses the format strings and validates them according to the expected format.

Right now this support is limited to direct library calls ( string.format("%.2f", ...) and literal strings used in these calls - we may lift some of these limitations later to include e.g. support for constant locals.

Additionally, if you have type checking beta enabled, string.format will now validate the argument types according to the format string to help you get your %d s and %s es right.

Format string analysis has some issues:

Treats using %1 with no captures as invalid, when it’s valid.

Treats an empty set [] or [^] as valid, the first character isn’t considered for closing the set (so something like []] is valid).

Same with frontier patterns %f[] or %f[^]

Treats %z and %Z as invalid

Some odd cases with ranges

% is part of the range in this case, shouldn’t consider the next character

And these too

Similar thing here, probably also related to accepting empty sets but this is valid

Q here should generate a warning, as it’s an invalid character class in a position to be used as a character class.

Or better yet, it should warn for creating a range which uses character classes, considering the manual states:

The interaction between ranges and classes is not defined. Therefore, patterns like [%a-z] or [a-%%] have no meaning.

Odd interaction with nested captures, the first is invalid while the second is valid

Doesn’t check if integral size is within [1,16]

Doesn’t check for anything to be after X

[anon66957764]

I saw the string.format analysis and that gave me an idea. What if Luau supported compile-time/runtime analysis things like Rust does?

Like, for example, defining a Luau function that actually gets executed during analysis that can access type information / tokens / etc. and can provide warnings/errors (squigglies) with a custom message.

But, of course, the Luau analyzer is probably not written in Luau which would make it very difficult and expensive to translate all of its internal state into a straightforward Luau data structure, so this will probably never be a thing… :(

Things like communicating what types the function takes based on a string you pass to it would be great to support generically rather than as a special feature for string.format.