Allow us to get the script's AST and Lexeme table

[metatablecatmaid (metatablecat)]

Release Notes for 567

My issue here is that unless there’s a Luau implementation thats close to the lexer/AST used, people who want to use my plugin would have to install and run a third-party program which can be Wouldn’t it be easier and safer here to expose methods in Studio that could work with immutable copies of the AST and Lexeme tokens

As a Roblox developer, it is currently too hard to effectively utilise the two callbacks under ScriptEditorService.

This is because, as the quoted post above shows, it is not possible to easily learn about the script’s AST table. Because of this, the callbacks can only do a few basic tasks, such as snippets. This becomes a bigger problem with the new RegisterScriptAnalysisCallback function that was recently released.

This callback is effectively useless without any way of quickly parsing the script into usable tokens (Lexemes or AST names).

Instead, if we could derive a script’s shape from the already generated AST table, we could better learn how to parse and handle our own analysis callbacks

-- script is print("Hello World")
ScriptEditorService:RegisterScriptAnalysisCallback(function(analysis)
  local document = ScriptEditorService:FindScriptDocument(analysis.Script)
  print(document:GetLexemes()) --> Lexemes of shape {AstName (print), open bracket, string, close bracket}
-- also document:GetAst() maybe but idk what you'd return here
end)

If Roblox is able to address this issue, it would improve my development experience because I could learn about a script’s shape that could then be easily used in the callbacks.

[WheretIB]

I agree that it would’ve been useful to provide that in these APIs.

We did consider it before adding these functions, but decided not to put it on the roadmap right now.
I also don’t think that there was an announcement for these additions, as we would’ve talked about this limitation there.

[Elttob (dphfox)]

This this so much this. I have wanted AST access for years. The lack of it has stopped me from developing so many plugins and developer tools in the past because it is a pain in the ass to have to build your own entire parsing stack from scratch just to do basic context-aware manipulations.