Lexer for RBX Lua

I was looking for a lexer which was not part of a lua compiler. I was thinking of writing my own and started even doing so but thank god I found this. Now I just need to write a parser for it. But thank you! :smiley:

This is very useful! This might seem like a stupid question, but why use _ before some properties of the nav object (_RealIndex, _UserIndex and _ScanThread) but not in others (Source, TokenCache)?

Also in the Api & Usage section there’s a typo.

Fixed, thanks for catching that!

It’s general practice to use a _ prefix to denote “private” values. That is to say, when you use the nav object, you shouldn’t ever be directly interacting with nav._ScanThread, because you risk breaking the nav if you alter it. Those values as used internally by the object itself, but aren’t recommended to be used by the higher level.
Source and TokenCache are “public” because I’ve found it’s useful to check what source is currently active and to directly index a token at a specific place (ex: checking the first token).

4 Likes

Hey guys! I’ve continued working on this!

I’ve refactored it (no more coroutines or return function list) and added contextual tokenizing (os.clock() will highlight clock as a builtin, and myThing.spawn() will no longer highlight spawn as the builtin function).

I’ve also released a highlighter module to go with this lexer! It uses RichText labels to highlight the code based on the tokens the lexer spits out.

Rather than continue to post into this obscure thread, I’ve opened a GitHub repository for it:

8 Likes

Lexer is breaking down code into different segments, often referred to as tokens according to token types (example keywords, operators, strings, brackets) and other criteria. This is used to parse, analyze, highlight and evaluate code syntax.

I like it. This lexer version is much more cleaner and understandable, I really didn’t like the coroutines that you were previously using. Hopefully with that change you will be able to add the newer and more complex syntax highlighting for methods, function names, properties and others that Luau has got in Studio. Good job!

Also, since it’s a GitHub repo, will other developers be able to contribute to this project as well?

2 Likes

Contribution is welcome! I will likely be adding more features as well. You can also open GitHub issues with suggestions/requests/problems!

4 Likes