Luau Recap: October 2020

[zeuxcg]

Luau is our new language that you can read more about at https://roblox.github.io/luau ; we’ve been so busy working on the current projects that we didn’t do an update in September, so let’s look at changes that happened since August!

Many people work on these improvements, with the team slowly growing - thanks @Apakovtac, @EthicalRobot, @fun_enthusiast, @machinamentum, @mrow_pizza and @zeuxcg!

Types are very close

We’ve been in beta for a while now, but we’re steadily marching towards getting the first release of the type checker, what we call “types v0”, out of the door. It turns out that we’ve substantially underestimated the effort required to make the type system robust, strike the balance between “correct” and “usable” and give quality diagnostics in the event we do find issues with your code

Because of this, we’re changing the original plans for the release a bit. We’re actively working on a host of changes that we consider to be part of the “v0” effort, and when they are all finished - which should happen next month, fingers crossed - we’re going to be out of beta!

However, by default, on scripts with no annotations, we won’t actually activate type checking. You would have to opt into the type checking by using --!nonstrict or --!strict, at the top of each script. We are also going to open the second beta, “All scripts use non-strict mode by default” or something along these lines.

This is important because we found that our non-strict mode still needs some more work to be more tolerant to some code that occurs commonly in Roblox and is correct, but doesn’t type-check. We’re going to evaluate what changes specifically are required to make this happen, but we didn’t want the extra risk of a flood of reports about issues reported in existing code to shift the release date in an unpredictable fashion.

To that end, we’ve been working on Lots and Lots and Lots and Lots and Lots of changes to finish the first stage. Some of these changes are already live and some are rolling out; the amount of changes is so large that I can’t possibly list the up-to-date status on each one as these recaps are synthesized by the human who is writing this on a Friday night, so here’s just a raw list of changes that may or may not have been enabled:

• Strict mode is now picky about passing extra arguments to functions, even though they are discarded silently at runtime, as this can hide bugs
• The error message about using a : vs . during type checking is now much more precise
• Recursive type annotations shouldn’t crash the type checker now, and we limit the recursion and iteration depth during type checking in a few cases in general in an effort to make sure type checker always returns in finite time
• Binary relational operators (< et al) are now stricter about the argument types and infer the argument types better
• Function argument and return types are now correctly contra- and co-variant; if this looks like gibberish to you, trust me - it’s for the best!
• Fixed a few problems with indexing unions of tables with matching key types
• Fixed issues with tracing types across modules (via require) in non-strict mode
• Error messages for long table types are now trimmed to make the output look nicer
• Improve the interaction between table types of unknown shape ({ [string]: X }) and table types of known shape.
• Fix some issues with type checking table assignments
• Fix some issues with variance of table fields
• Improve the legibility of type errors during function calls - errors now point at specific arguments that are incorrect, and mismatch in argument count should clearly highlight the problem
• Fix types for many builtins including ipairs, table.create, Color3.fromHSV, and a few others
• Fix missing callbacks for some instance types like OnInvoke for bindables (I think this one is currently disabled while we’re fixing a semi-related bug, but should be enabled soon!)
• Rework the rules under which globals are okay to use in non-strict mode to mostly permit valid scripts to type-check; strict mode will continue to frown upon the use of global variables
• Fix a problem with the beta where two scripts with identical names would share the set of errors/warnings, resulting in confusing error highlights for code that doesn’t exist
• Improve the legibility of type errors when indexing a table without a given key
• Improve the parsing error when trying to return a tuple; function f(): string, number is invalid since the type list should be parenthesized because of how our type grammar is currently structured
• Type checker now clearly reports cases where it finds a cyclic dependency between two modules
• Type errors should now be correctly sorted in the Script Analysis widget
• Error messages on mismatches between numbers of values in return statements should now be cleaner, as well as the associated type mismatch errors
• Improve error messages for comparison operators
• Flag attempts to require a non-module script during type checking
• Fix some cases where a type/typeof guard could be misled into inferring a non-sensible type
• Increase the strictness of return type checks in strict mode - functions now must conform to the specified type signature, whereas before we’d allow a function to return no values even in strict mode
• Improve the duplicate definition errors to specify the line of the first definition
• Increase the strictness of binary operators in strict mode to enforce the presence of the given operator as a built-in or as part of the metatable, to make sure that strict mode doesn’t infer types when it can’t guarantee correctness
• Improve the type errors for cyclic types to make them more readable
• Make type checker more friendly by rewording a lot of error messages
• Fix a few crashes in the type checker (although a couple more remain - working on them!)
• … I think that’s it?
• …edit ah, of course I forgot one thing - different enums that are part of the Roblox API now have distinct types and you can refer to the types by name e.g. Enum.Material; this should go live next week though.

If you want to pretend that you’ve read and understood the entire list above, just know that we’ve worked on making sure strict mode is more reliably reporting type errors and doesn’t infer types incorrectly, on making sure non-strict mode is more forgiving for code that is probably valid, and on making the type errors more specific, easier to understand, and correct.

Type syntax changes

There’s only two small changes here this time around - the type syntax is now completely stable at this point, and any existing type annotation will continue parsing indefinitely. We of course reserve the right to add new syntax that’s backwards compatible

On that note, one of the small changes is that we’ve finally removed support for fat arrows (=>); we’ve previously announced that this would happen and that thin arrows (->) are the future, and had warnings issued on the legacy syntax for a while. Now it’s gone.

On a positive note, we’ve added a shorter syntax for array-like table types. Whereas before you had to use a longer { [number]: string } syntax to declare an array-like table that holds strings, or had to define an Array type in every. single. module. you. ever. write. ever., now you can simply say {string}! This syntax is clean, in line with the value syntax for Lua table literals, and also was chosen by other research projects to add type annotations to Lua.

(if you’re a monster that uses mixed tables, you’ll have to continue using the longer syntax e.g. { [number]: string, n: number })

Parallel Luau is in progress!

We’re actively working on our first beta release of parallelism support. We plan to ship something by the end of the year in Studio as a beta - we want to keep iterating on some possibly-breaking changes, so this will be a beta-only release, but we’re excited to share our plans with you.

The release will also be a small part of the full vision - the full technical specification for this is 26 pages (this was hard to write…), and we aren’t at the point where we implemented the entire thing - not even close Our plan is to gradually expand what you can do in parallel over the course of the next year (in addition to releasing this outside of beta of course!), while maintaining safe sandboxing (if you manage to crash the engine by running code in parallel, that’s always a bug on our end).

For people who are daring and adventurous, I’ll leave a few hints and you can try to explore this on your own: ParallelLua, Actor, Heartbeat:connectParallel, TaskLibrary, task.synchronize, task.desynchronize. Connecting the pieces is up to you until we actually release a beta

Library changes

There’s only a few small tweaks here this time around on the functionality front:

• utf8.charpattern is now exactly equal to the version from Lua 5.3; this is now possible because we support \0 in patterns, and was suggested by a user on devforum. We do listen!
• string.pack now errors out early when the format specifier is Way Too Large. This was reported on dev forum and subsequently fixed. Note that trying to generate a Moderately Large String (like, 100 MB instead of 100 GB) will still succeed but may take longer than we’d like - we have a plan to accelerate operations on large strings substantially in the coming months.

Performance improvements

We were super focused on other things so this is very short this time around. We have a lot of ideas here but they are waiting for us to finish some other large projects!

• Method calls on strings via : are now ~10% faster than before. We still recommend using fully-qualified calls from string library such as string.foo(str), but extra performance never hurts!
• Speaking of string methods, string.sub is now ~20% faster than before with the help of voodoo magic.

Miscellaneous fixes

There were a few small fixes that didn’t land into any specific category that I wanted to highlight:

• In some rare cases, debug information on conditions inside loops have been fixed to stop debugger from incorrectly suggesting that the current line is inside a branch that wasn’t taken. As usual, if you ever see debugger misbehaving, please file bugs on this!
• Code following assert(false) is now treated as an unreachable destination from the linting and type checking point of view, similarly to error calls.
• Linting support for various format strings has been greatly improved based on fantastic feedback from @Halalaluyafail3 (thanks!).

Ok, phew, that’s what I get for skipping a month again. Please don’t hesitate to report bugs or suggestions, here or via separate posts. Due to our usual end-of-year code freeze there’s going to be one more recap at the end of the year where we will look back at 2020 and take a small peek into the future.

[loueque (paperbagger)]

Such a long but great informational topic for users scripting in Lua/Luau, In do have a question:

@zeuxcg

On that note, one of the small changes is that we’ve finally removed support for fat arrows ( => ); we’ve previously announced that this would happen and that thin arrows ( -> ) are the future, and had warnings issued on the legacy syntax for a while. Now it’s gone.

May I ask the problem with => and why did it get converted to ->? I feel like this might be confusing to some developers and using the = / == feels easy to understand and have inside a script.

@zeuxcg

On a positive note, we’ve added a shorter syntax for array-like table types. Whereas before you had to use a longer { [number]: string } syntax to declare an array-like table that holds strings, or had to define an Array type in every. single. module. you. ever. write. ever., now you can simply say {string} ! This syntax is clean, in line with the value syntax for Lua table literals, and also was chosen by other research projects to add type annotations to Lua.

(if you’re a monster that uses mixed tables, you’ll have to continue using the longer syntax e.g. { [number]: string, n: number } )

Coolio! I’m somewhat the “monster” and “shorter syntax” scripter myself and will find this helpful in the future with Luau live or not and some more improvements with the categories above and below the quotated message.

@zeuxcg

The error message about using a : vs . during type checking is now much more precise

Interesting, the war with : and . (O_O) still goes on, but having it be more precise seems nice to have in Luau.

[zeuxcg]

You’re confusing this with <= / >= (which is a comparison operator). => was previously used in function type declarations, e.g. local foo: (number) => string = tostring, but we changed it to -> to be more readable.

[DavidNet22 (Dvd)]

@zeuxcg

Types are very close

Visible excitation*

Such an amazing month (or months) for the Luau language. We had some inconvenients with the data loss of last week, but overall a good month

[Dekkonot]

Parallel Lua you say? I can’t wait to accidentally make my code slower by multithreading it.

The array syntax is a welcome change.

Currently the main thing that’s stopping me from using typed Lua is the lack of type assertions; I know it’s not possible to have as (even though I would very much like it to be ) but there has to be some solution since right now the only way to assert something about a value is by adding new code – something that’s necessary anyway for some cases but for others is just added branching or maneuvering, all for the sake of script analysis.

Specifically, we need a way to assert that a table is of a certain type (I have no good suggestions for the syntax for this), and we need a way to declare that a value isn’t nil without adding a guard against it. TypeScript has ! for that and I still think it would be a good choice for Luau.

Otherwise, I’m happy with typed Lua in its basic form right now. Some way to explicitly type metatables would be good, but that’s not necessary for most code and is workable when it is required.

[zeuxcg]

Yup, these all make sense and we’re aware of them, as well as of some other low hanging fruit (e.g. the lack of a way to model variadics with an ascribed type actually causes issues for some of our builtins as well as for user code); realistically we’re going to continue working on types throughout next year, with a few other significant changes planned, but we needed to make a call as to when to call the work we’re doing now done

After we go out of beta we plan to continue making incremental changes to make the type system and the language itself more awesome, it just won’t be part of a beta program (short of the enablement of nonstrict type checking for all scripts by default, which requires a few other things we haven’t quite done because there the requirement is “don’t show scary type errors to users unless we’re pretty sure the code will fail at runtime anyway”)

[LifeDigger (Life)]

So the games made in Lula will still work correct because Lua because it is compatible right? I am just a little concerned that there may be some bugs on games due to this update. Also is there any point on learning this new language if we already know Lua?

[plasma_node]

@zeuxcg

Parallel Luau is in progress!

This is just awesome. Many of the cool things I’ve tried to do in roblox just can’t be done on one core. In addition to making fun things like CPU ray tracers faster, this can be a significant performance gain for certain types of games.

I have two questions:

1. How many cores will we be able to use? Is there a limit like 6 or 8 or just how many cores the computer has?

2. Are we directly able to assign threads to a certain core or does the task library/some api handle that?

[focasds]

Can you guys add one day if possible a way of highlighting something as a built-in function? Something like this:
It will be very helpful, at least for me and those people that likes to highlight their code and keep it organized and understandable. I see that you guys are adding those comments to prevent the editor from doing some stuff, will it be nice to add this type of thing? (The code is just an example by the way) My code actually looks pale currently.

[codes4breakfast (Lucas)]

The new shorthand array syntax is very much appreciated and will definitely make a lot of code more succint. Thank you

On a side note, can we expect enum support for this initial release? I realise it’s probably a bit late to ask for this now but it’s one of the few things I can’t annotate precisely in my code, settling with just specifying EnumItem as the type and dealing with any wrong enums passed as they come by (Enum.InputType and Enum.UserInputType are some I catch myself swapping often, for instance).
EDIT: Oh whoops, I missed that edit there. Glad to know that enums were given some love and that this is no longer a problem. Thanks again

Other than that, the lack of type assertions mentioned by others also complicates code in situations where I know something is not nil but the type checker does not, forcing me to resort to redeclaring variables to cast types to any and back (or just using any everywhere altogether). This comes up more often than I expected and is the major issue blocking me from using typing more proactively, so anything on this front would be immensely helpful.

Thank you for the amazing work on Luau so far. It has saved me from myself many times already.

[Hexcede]

Luau does not change anything about lua’s syntax or its behavior, its just an extension of lua with better performance, and additional features (like type checking).

As for the recap, I am very hyped! I’m once again excited to optimize my compression algorithm, unfortunately of which I’ve hardly had time to work on (yay school, Covid, and 2020)

The hints make me very curious. Here is what I guess that they will do with my limited understanding of how the heck multithreading tends to work:
ParallelLua - For this, I have no idea what this could be… Other than a library. Based on the wording, I think this will have constructors or helpers, e.g. ParallelLua.new or ParallelLua.spawn, but, I’m not sure what this may create. Perhaps the Actor could be related to its return type but I more so suspect it will aid in the creation of several types.
Actor - I think this could potentially be a data type, or possibly a class of Instance which would facilitate certain parallel behaviors between scripts. Basically, scripts could signal each other through the Actor and unlike events everything would be happening on different CPU threads. It’ll be interesting though to see what all this may have.
Heartbeat:connectParallel - This very clearly refers to the Heartbeat event of RunService, and, I think very clearly indicates that all events could be connected so that their callbacks are run parralel - Or perhaps instead we would pass tasks?
TaskLibrary - This very clearly will be used in the creation of tasks, of which I assume the type or class name will be Task?
(These two I have the least confidence on)
task.synchronize - This could potentially be used to synchronize two or more tasks together so that you could guarantee one task will process in unison with another, perhaps to stop race conditions so if one task is running slow the other one won’t race ahead through your code.
task.desynchronize - This could do the opposite, allowing the task to process whenever it felt like even if another task is processing faster, so if one is running slower it would fall behind.

[Fluffmiceter]

Very happy to see this post as a whole, I said in the 2021 RDC speakers survey that I wanted to hear more Luau progress, looks like Christmas came early! I also explicitly mentioned I wanted to hear more about parallel Lua, so it’s a cool coincidence that this got mentioned now…

Being able to run parallel processes on heartbeat should be a fairly simple and elegant way to enable multithreading in a very practical/easy to implement way. i.e if you have some AI that runs for each zombie in a game, and that AI script runs off of heartbeat, you could easily split that work to run on separate threads with a “Heartbeat:connectParallel” (I assume that’s what that means).

I assume there is a “task” object which is constructed with a function as a parameter, and then you can “synchronize” two tasks together where these two linked tasks basically behave as one yield in the lua thread, and the yield stops once the execution of both tasks finishes; luau would decide whether or not to place those tasks on multiple hardware threads based on availability?

[DataBrain (Amber)]

Awesome! I can’t wait until we get to V1 of typed luau.

It looks like OOP is also (kind of) reasonable to use now when relying mostly on type inference and not-so-sugary syntax. However, the type checker seems to not like __index calls outside of the class code. This seems like a regression, as before I was able to call functions defined in a table’s __index metamethod:

game.ReplicatedStorage.Person (no script analysis errors):

--!strict

local Person = {}
Person.__index = {}

function Person.new(_name: string, _dob: number)
	local self = {}
	setmetatable(self, Person)
	
	self.name = _name
	self.dateOfBirth = _dob
	
	return self
end

-- Exported as Person.Class for convenience
export type Class = typeof(Person)
-- Exported as Person.Object for convenience
export type Object = typeof((function() local _name: string, _dob: number return Person.new(_name, _dob) end)())

function Person.__index.PrintName(self: Object): ()
	print(self.name)
end

function Person.__index.GetAge(self: Object): number
	return math.floor((os.time() - self.dateOfBirth) / 365.25 / 24 / 60 / 60)
end

return Person

game.ServerScriptService.Script (script analysis error on the last line):

--!strict

local Person = require(game.ReplicatedStorage.Person)

local bob = Person.new("Bob", 0)

print(bob:GetAge())

Output when running the game (works as expected):

Yes, it’s really been 50 years since the Epoch. Wow.

I’m also still waiting for the day we’ll be able to import exported types from a module without requiring that module, in order to avoid some require deadlocks.

Overall though, I’m glad to see this project becoming more and more usable in production code!

[iiNemo (nick)]

After countless hours working on a project from vscode and importing to Roblox, I’ve come across one of the oddest bugs. Every time I edited code in a specific script it crashed studio. Now with it being over 700 lines I immediately went straight to Netflix. Hours later I built up the courage to pinpoint the problem and ended up finding something.
what.rbxm (824 Bytes)

By removing the comment from that line it will immediately crash your studio and it seems to happen only when type checking is on. Assuming it has some kinda cyclic problem, but I just can’t seem to figure out why.

Didn’t mean to reply directly to DataBrain although I do love some of his ideas

I’m also still waiting for the day we’ll be able to import exported types from a module without requiring that module, in order to avoid some require deadlocks.

[Dekkonot]

EDIT: Haha nope I was wrong, go look at the actual post about it here

Old post here

While I am by no means an expert on multithreading, I believe I’ve gotten Parallel Lua working (at least a little bit). Testing was done on sitetest3, but this stuff exists on sitetest1 so it’s probably fine to use that. For anyone curious, these are my observations:

• ParalellLua and TaskLibrary are FFlags that you should turn on to mess with Parallel Lua
• Actor is a new Instance that acts as a container for parallel code (@zeuxcg is there a reason it inherits from Model and isn’t its own thing like Folder?)
  • Scripts which use ConnectParallel must be parented under an Actor to run
• ConnectParallel is a new method on RBXScriptConnection
  • this method runs the passed function in parallel but is otherwise normal
• task is injected into the environment of scripts that are parented under Actor instances
  • it contains synchronize, which locks the DataModel to the calling coroutine, and desynchronize which unlocks it
  • if synchronize isn’t called in a coroutine, it will throw an error when trying to access the DataModel

I ran this code to test it:

local counter = Instance.new("IntValue")
counter.Name = "Counter"
counter.Parent = workspace

for i = 1, 10 do
    local connection
    connection = game:GetService("RunService").Heartbeat:ConnectParallel(function()
        task.synchronize()
        counter.Value += 1
        task.desynchronize()
        connection:Disconnect()
    end)
end

On my not-very-good CPU, that set counter.Value to 65 instead of 10 (which is what you might expect looking at the code).

My conclusion: multithreading is hard but the results are promising, and I’m looking forward to actual documentation instead of just figuring it out as I go along.

[TheBristlefrost (Bristlefrost)]

The biggest bottleneck in Luau for me at least would be the lack of as keyword. Let’s say I have a variable with a type sometype | nil. At some point in my code the type checker will complain about no key somekey found in table sometype | nil although I’m sure it’s not nil but the type checker isn’t.

Other programming languages have type assertion, casting etc, why doesn’t Luau?

[Oseday]

Will we be able to update instance properties in parallel? Or will the properties have a mutex lock on them? I would really like for the code running in the parallel threads to not lock resources (including Lua tables) while reading them or writing to them. This part is very crucial to be able to benefit from parallel computing.

Any plans for compute shaders?

[Hydrinated]

Nooo! So does this mean that all the roblox scripting tutorials on youtube won’t work? Now I’m really not gonna know how to code…