LuauParser ( full v0.710 )

[vantoanvh]

LuauParser

GitHub | Download | Documentation

A modern, fully featured Luau parser written entirely in Luau
Fully type-checked with the New Type Solver and optimized for high performance.

Introduction

---

Only compatibles with New Type Solver.

A modified, yet complete port of Parser.cpp , providing both AST (Abstract Syntax Tree) & CST (Concrete Syntax Tree).
To stay true to the original implementation, the CST does not include low-level trivia such as whitespace.

This Luau parser is primarily intended for plugin development and for building Luau compilers or tooling.
It provides detailed syntax through both AST and CST, making it well suited for advanced language tooling such as linters, formatters, highlighter, refactoring tools, or even a full luau compiler/transpiler.

NOTE: I wrote this with the help of AI (because I don’t want to spend time porting an entire codebase to another language by hand, yes it is easy to make AS SAME AS time consuming). Truthfully, just want to say that I don’t want myself to be credited all the work here even though I took tens of hours fixing AI slops and debug code by hand.

Performance

---

This parser is fast because it uses a simple singleton-style procedural design. It just resets its internal variables and runs, avoiding the overhead of more complex architectures.

Purpose & Code usage

---

This Luau parser is designed for plugin development and language tooling, providing detailed syntax data via AST and CST for linters, formatters, highlighters, refactoring tools, and compilers.

local Parser = require(path.to.LuauParser)

local success, result = Parser.parse('local foo: (string, ...number) -> (...any)')

if success then
	print(result.root) -- do anything you want
else
	warn(result.errors)
end

Question & Answer

---

1. This LuauParser compare to luaup, which is better?

• PROS: This parser supports syntax features up to v0.710, better error handling.

• CONS: luaup seems to offer simplicity, readability but less features, also a little faster.

2. Does it support autocomplete?

• Yes, it is!

Future notes

---

I tried to make a modern Luau Compiler in near future, but that would be a long time to make.
Everything is possible, and hoping it will be faster than LuauCeption.

Help & Feedback!

---

Review any inaccuracies or misinformation in the parser documentation and reply me if you can, because the documentation was made by Codex which I don’t really trust it at making them.

[23sinek345 (Dr_Sinek)]

Would be great if you provided benchmarks with LuauCeption

[vantoanvh]

LuauCompiler is still just a prototype and doesn’t run properly yet, so I can’t provide benchmarks. However, it may be much faster than LuauCeption because LuauCeption converts WASM to Luau, which adds a lot of overhead, while LuauCompiler will be an optimized Luau implementation.

If I am correct, do you meant luaup? so it’s a little bit slower than luaup as far as I tested.
Here’s some benchmark:

File	LuauParser	luaup
jecs.luau	28.95 ms	24.04 ms
LuauParser source code	32.95 ms	25.71 ms

[Lewin4]

I saw your repo a while ago and spent too long trying to figure out who this mysterious hero was that randomly released this awesome library. For a while I’ve been searching for a Luau parsing library that maintains similarity with the reference parser implementation, and this seems to be perfect; most other parsers deviate from the reference implementation significantly or tend to omit features their authors don’t need.

I’ve also been trying to write my own Luau parser, though have previously struggled with correctness and conformance. Having your version to reference made this a ton easier, and I ended up porting it to Go to see if I can improve its performance further (even though a lot of Luau optimisations don’t apply to many compiled languages). Previously I’ve had to invoke luau-ast on the command line and parse its output into JSON – this library is orders of magnitude faster than my old method, especially so on large files (I benchmarked with LuauCeption too ;). I’m absolutely planning on creating a compiler or similar using it.

Are there any plans to update the library to v0.711 or later? I’d really like to be able to parse const. Also let me know if you’re looking for contributors or testers as I would be delighted to help!

[KrisstalDev (Krisstal)]

Hey, I’m making a game where you have to code, and I’m currently making a code editor.
I tried to make the autocomplete system and I did find a way to make it, but I don’t understand all of the nodes, so can you help me out a bit? It’s just one module that analyzes the code using your parser

[vantoanvh_dev]

You need to make a tree walker for the parser output

For example, when the walker sees a local x, it checks if the cursor is after that declaration. If yes, x can be suggested.

When it sees a function, loop, or block, it checks whether the cursor is inside that scope. If the cursor is inside, the walker enters that block and scans its children too.

I have a hand-make code for that, AND THIS IS NOT FULL AUTOCOMPLETE, I just make it for learning, and it works! It not support complex autocomplete like Module or Types and such, just simple one like variables, functions, scoped block, global function, for i loop, for in loop.

This code is pretty good, and you could upgrade it though

-- A just simple demo about autocomplete, I did not make full one, just some important ones

-- Require the Luau parser module.
local Parser = require("@game/ReplicatedStorage/LuauParser")

-- Current cursor position inside the editor.
-- IMPORTANT:
-- LuauParser locations are 0-indexed:
--   line 0 = first line
--   column 0 = first column
--
-- This means:
--   line = 7
--   column = 3
-- refers to the 8th line, 4th character. which is 
local CurrentLine, CurrentColumn = 7, 3

local source = [[local x, y = 6, 7
	
const function Abc(arg)
	local InsideVariable = 2

	for var1, var2, var3, var4 in ABCD do
		for index = 1, 100, 2 do
			-- we are here.
		end
	end
end

function Global()

end
]]

-- Checks whether a variable declaration is already in scope at the cursor.
-- We should detect if the cursor ARE after the variable declaration.
local function IsVariableInScopeAtCursor(nodePosition)
	-- If the cursor is on a later line than the declaration,
	-- then the variable is definitely in scope.
	if CurrentLine > nodePosition.line then
		return true
		-- If the cursor is on the exact same line,
		-- check whether the cursor is after the declaration column.
	elseif CurrentLine == nodePosition.line then
		return CurrentColumn > nodePosition.column
	end
end

-- Checks whether the cursor is currently INSIDE a block body.
-- Detect like if we are inside a functions, a loops, or any scoped body
local function IsCursorInsideBody(bodyLocation)
	-- Ending position of the body.
	local end_ = bodyLocation.end_

	-- Starting position of the body.
	local begin = bodyLocation.begin

	-- Cursor must be:
	--   AFTER or ON the begin position
	--   AND
	--   BEFORE the end position
	return (
		(CurrentLine > begin.line
			or (CurrentLine == begin.line and CurrentColumn >= begin.column))
	)
		and (
			(CurrentLine < end_.line
				or (CurrentLine == end_.line and CurrentColumn < end_.column))
		)
end

-- Recursively walks through a block of AST nodes
-- This is currently just supports: local, global function, local function, for in and for i loop, and scoped block.
-- I didn't add global function because I am too lazy...
local function walksThoughBlock(ReturnedTable, Block)
	for _, node in Block do
		-- Local variable like `local x,y = 6,7`
		if node.kind == "StatLocal" then
			-- Only include variables declared BEFORE the cursor.
			if IsVariableInScopeAtCursor(node.location.end_) then
				-- node.vars contains all declared variables.
				for _, var in node.vars do
					table.insert(ReturnedTable, {
						Name = var.name,
						Type = "Variable",
					})
				end
			end
		end

		-- Local function like `local function Main()`
		if node.kind == "StatLocalFunction" then
			local func = node.func
			local block = func.body

			-- Only continue if cursor is inside this function body.
			if IsCursorInsideBody(block.location) then

				-- Recursively scan inside the function body.
				walksThoughBlock(ReturnedTable, block.body)

				-- Add the function name itself to autocomplete.
				table.insert(ReturnedTable, {
					Name = node.name.name,
					Type = "Function",
				})

				-- Add all function arguments as variables.
				for _, arg in func.args do
					table.insert(ReturnedTable, {
						Name = arg.name,
						Type = "Variable",
					})
				end
			end
		end
		
		-- Global function like `function Global()`
		if node.kind == "StatFunction" then
			local func = node.func
			local block = func.body

			-- Global functions are always visible,
			-- so insert immediately.
			table.insert(ReturnedTable, {
				Name = node.name.name,
				Type = "Function",
			})

			-- If cursor is inside the function body:
			if IsCursorInsideBody(block.location) then

				-- Recursively process nested scopes.
				walksThoughBlock(ReturnedTable, block.body)

				-- Add function arguments.
				for _, arg in func.args do
					table.insert(ReturnedTable, {
						Name = arg.name,
						Type = "Variable",
					})
				end
			end
		end
		
		-- Numeric for loop like `for i = 1, 10 do`
		if node.kind == "StatFor" then
			local block = node.body

			-- Only process loop variables if cursor is inside loop body.
			if IsCursorInsideBody(block.location) then

				-- Recursively process nested contents.
				walksThoughBlock(ReturnedTable, block.body)

				-- Add loop variables.
				table.insert(ReturnedTable, {
					Name = node.var.name,
					Type = "Variable",
				})
			end
		end
		
		-- Generic for loop like `for k, v in Table do`
		if node.kind == "StatForIn" then
			local block = node.body

			-- Only process variables if cursor is inside the loop.
			if IsCursorInsideBody(block.location) then

				-- Recursively process nested contents.
				walksThoughBlock(ReturnedTable, block.body)

				-- Add all iterator variables.
				for _, var in node.vars do
					table.insert(ReturnedTable, {
						Name = var.name,
						Type = "Variable",
					})
				end
			end
		end
		
		-- Block scope like `do end`
		if node.kind == "StatBlock" then
			walksThoughBlock(ReturnedTable, node.body)
		end
	end

	return ReturnedTable
end

local success, result = Parser.parse(source)

-- We get all autocompletes values EVEN if error parsing, just like all the IDEs
local AllAutocompletes = walksThoughBlock({}, result.root.body)

And it prints correct

[KrisstalDev (Krisstal)]

I’ve made a recursive search so it first searches for the closest node to the cursor position, then it saves the path and goes through it and collects variables
After that it processes every variable, global and outputs their types.

Though it’s still not complete. I’ve spent 2 days trying to figure everything out, and I didn’t write the code for member access yet, but it kinda works

local module = {}

-- stuff in replicated storage
local SHARED = game.ReplicatedStorage.Shared
local MLuauParser = require(SHARED.Modules.LuauParser)
local GEnvironmentTypes = require(SHARED.Global.EnvironmentTypes)
local GScriptGlobals = require(SHARED.Global.ScriptGlobals)


local function calculateSimilarity(text1:string, text2:string)
	text1 = text1:lower()
	text2 = text2:lower()
	
	local len1 = #text1
	local len2 = #text2
	
	local matrix = {}
	
	for i = 0, len1 do
		matrix[i] = {[0] = i}
	end
	
	for j = 0, len2 do
		matrix[0][j] = j
	end
	
	for i = 1, len1 do
		for j = 1, len2 do
			local cost = text1:sub(i, i) == text2:sub(j, j) and 0 or 1
			
			matrix[i][j] = math.min(
				matrix[i-1][j] + 1,
				matrix[i][j-1] + 1,
				matrix[i-1][j-1] + cost
			)
		end
	end
	
	local maxLength = math.max(len1, len2)
	if maxLength == 0 then return 1 end
	
	local distance = matrix[len1][len2]
	return 1 - distance / maxLength
end

local function checkLocation(line:number, column:number, loc:{})
	if line < loc.begin.line or line > loc.end_.line then return false end
	if line == loc.begin.line and column < loc.begin.column then return false end
	if line == loc.end_.line and column > loc.end_.column then return false end
	
	return true
end

local function findDeepestNode(node:{[string]:any}, line:number, column:number, path:{{[string]:any}})
	if not node or not node.location then return nil end
	
	if not checkLocation(line, column, node.location) then
		return nil
	end
	
	local deeperNode:{[string]:any}? = nil
	
	table.insert(path, node)
	
	if node.kind == "StatBlock" and node.body then
		for _, stmt in pairs(node.body) do
			deeperNode = findDeepestNode(stmt, line, column, path)
			
			if deeperNode then break end
		end
	elseif node.kind == "StatLocal" then
		if node.values then
			for _, expr:{} in pairs(node.values) do
				deeperNode = findDeepestNode(expr, line, column, path)
				
				if deeperNode then break end
			end
		end
	elseif node.kind == "ExprCall" then
		deeperNode = findDeepestNode(node.func, line, column, path)
		
		if not deeperNode and node.args then
			for _, arg in pairs(node.args) do
				deeperNode = findDeepestNode(arg, line, column, path)
				
				if deeperNode then break end
			end
		end
	elseif node.kind == "ExprIndexName" then
		deeperNode = findDeepestNode(node.expr, line, column, path)
	elseif node.kind == "StatExpr" then
		deeperNode = findDeepestNode(node.expr, line, column, path)
	elseif node.kind == "StatLocalFunction" then
		deeperNode = findDeepestNode(node.func, line, column, path)
	elseif node.kind == "ExprFunction" then
		deeperNode = findDeepestNode(node.body, line, column, path)
	elseif node.kind == "ExprIfElse" then
		if node.condition then
			deeperNode = findDeepestNode(node.condition, line, column, path)
		end
		
		if node.trueExpr and not deeperNode then
			deeperNode = findDeepestNode(node.trueExpr, line, column, path)
		end
		
		if node.falseExpr and not deeperNode then
			deeperNode = findDeepestNode(node.falseExpr, line, column, path)
		end
	elseif node.kind == "StatIf" then
		if node.condition then
			deeperNode = findDeepestNode(node.condition, line, column, path)
		end
		
		if node.thenbody and not deeperNode then
			deeperNode = findDeepestNode(node.thenbody, line, column, path)
		end
		
		if node.elsebody and not deeperNode then
			deeperNode = findDeepestNode(node.elsebody, line, column, path)
		end
	elseif node.kind == "ExprTable" then
		for _, v in pairs(node.items) do
			deeperNode = findDeepestNode(v, line, column, path)
			
			if deeperNode then break end
		end
	elseif node.kind == "List" or node.kind == "Record" or node.kind == "General" then
		if node.key then
			deeperNode = findDeepestNode(node.key, line, column, path)
		end
		
		if node.value and not deeperNode then
			deeperNode = findDeepestNode(node.value, line, column, path)
		end
	elseif node.kind == "StatWhile" then
		if node.condition then
			deeperNode = findDeepestNode(node.condition, line, column, path)
		end
		
		if node.body and not deeperNode then
			deeperNode = findDeepestNode(node.body, line, column, path)
		end
	elseif node.kind == "ExprIndexExpr" then
		if node.expr then
			deeperNode = findDeepestNode(node.expr, line, column, path)
		end
		
		if node.index and not deeperNode then
			deeperNode = findDeepestNode(node.index, line, column, path)
		end
	elseif node.kind == "StatReturn" then
		for _, v in pairs(node.list) do
			deeperNode = findDeepestNode(v, line, column, path)
			
			if deeperNode then break end
		end
	elseif node.kind == "ExprBinary" then
		if node.left then
			deeperNode = findDeepestNode(node.left, line, column, path)
		end

		if node.right and not deeperNode then
			deeperNode = findDeepestNode(node.right, line, column, path)
		end
	elseif node.kind == "ExprUnary" then
		deeperNode = findDeepestNode(node.expr, line, column, path)
	end

	return if deeperNode then deeperNode else node
end

local function getAccessContext(node:{[string]:any}, code:string)
	if not node then return nil end
	
	if node.kind == "ExprError" or node.kind == "StatError" then
		node = node.expressions[1] or {}
	end
	
	if node.kind == "ExprIndexName" then
		local op = string.char(node.op)
		local line = code:split("\n")[node.opPosition.line+1]
		local start = node.opPosition.column+2
		
		return {
			type = "Member",
			node = node.expr,
			original = node,
			isMethod = op == ':',
			name = line:sub(start, (line:find("[%p%s]", start) or #line+1)-1),
		}
	elseif node.kind == "ExprGlobal" then
		local line = code:split("\n")[node.location.begin.line+1]
		local start = node.location.begin.column+1
		
		return {
			type = "Global",
			node = node,
			name = line:sub(start, (line:find("[%p%s]", start) or #line+1)-1)
		}
	elseif node.kind == "ExprLocal" then
		local line = code:split("\n")[node.location.begin.line+1]
		local start = node.location.begin.column+1

		return {
			type = "Local",
			node = node,
			name = line:sub(start, (line:find("[%p%s]", start) or #line+1)-1)
		}
	end
end

local function getFuncType(func:{[string]:any})
	local argsStr = ""
	local returnStr = ""
	
	if func.returnAnnotation then
		for i, v in pairs(func.returnAnnotation.types) do
			if v.kind == "TypeReference" then
				returnStr ..= v.name
			else
				returnStr ..= "?"
			end
			
			if i ~= #func.returnAnnotation.types then
				returnStr ..= ", "
			end
		end
	end
	
	for i, v in pairs(func.args) do
		argsStr ..= v.name
		
		if v.annotation and v.annotation.kind == "TypeReference" then
			argsStr ..= ": "..v.annotation.name
		end
		
		if i ~= #func.args then
			argsStr ..= ", "
		end
	end
	
	return "("..argsStr.."): "..returnStr
end

local function getPossibleNamesNodes(node:{[string]:any}, name:string, path:{{[string]:any}}, envTypes:{[string]:string})
	local result:{{name:string, node:{[string]:any}, similarity:number}} = {}
	
	--[[for i, v in pairs(GScriptGlobals.All) do
		table.insert(result, {name = i, node = {kind = "GLOBAL", name = i}})
	end]]
	
	for i, v in pairs(envTypes) do
		table.insert(result, {name = i, node = {kind = "GLOBAL", name = i}})
	end
	
	--[[for _, v in pairs(GScriptGlobals.Keywords) do
		table.insert(result, {name = v, node = {kind = "KEYWORD", name = v}, type = "KEYWORD"})
	end]]
	
	local function checkNode(curNode:{[string]:any})
		if curNode == node then return end
		
		if curNode.kind == "StatBlock" then
			for _, v in pairs(curNode.body) do
				checkNode(v)
			end
		elseif curNode.kind == "StatLocal" then
			for _, v in pairs(curNode.vars) do
				v.kind = "VAR"
				
				checkNode(v)
			end
		elseif curNode.kind == "ExprFunction" then
			for _, v in pairs(curNode.args) do
				v.kind = "VAR"
				
				checkNode(v)
			end
		elseif curNode.kind == "VAR" then
			table.insert(result, {name = curNode.name, node = curNode})
		elseif curNode.kind == "StatLocalFunction" then
			curNode.name.kind = "VAR"
			
			table.insert(result, {name = curNode.name.name, node = curNode.name, type = getFuncType(curNode.func)})
		end
	end
	
	for i=#path-1, 1, -1 do
		local curNode = path[i]
		
		checkNode(curNode)
	end
	
	for _, v in pairs(result) do
		v.similarity = calculateSimilarity(v.name, name)
	end
	
	table.sort(result, function (v1, v2)
		return v1.similarity > v2.similarity
	end)
	
	print(node, result)
	
	return result
end

-- table types: detailed descriptions of types that are tables
-- env types: types of all provided globals (gets from server)
local function getPossibleType(node:{[string]:any}, envTypes:{[string]:string}, tableTypes:{[string]:{}})
	if node.kind == "ExprLocal" then
		if node["local"].annotation and node["local"].annotation.kind == "TypeReference" then
			local name = node["local"].annotation.name
			
			return tableTypes[name] or name
		else
			return "?"
		end
	elseif node.kind == "ExprGlobal" then
		if envTypes[node.name] then
			return tableTypes[envTypes[node.name]] or envTypes[node.name]
		else
			return "?"
		end
	elseif node.kind == "VAR" then
		if node.annotation and node.annotation.kind == "TypeReference" then
			local name = node.annotation.name
			
			return tableTypes[name] or name
		else
			return "?"
		end
	elseif node.kind == "GLOBAL" then
		if envTypes[node.name] then
			return tableTypes[envTypes[node.name]] or envTypes[node.name]
		else
			return "?"
		end
	end
end


function module.getHints(code:string, cursorLine:number, cursorColumn:number, envTypes:{[string]:any})
	local success, result = MLuauParser.parse(code)
	local path:{{[string]:any}} = {}
	local cursorNode = if result.root then findDeepestNode(result.root, cursorLine, cursorColumn, path) else nil
	
	if cursorNode then
		local context = getAccessContext(cursorNode, code)
		
		if context then
			local result:{{name:string, type:string|{[string]:string}}} = {}
			
			for _, v in pairs(getPossibleNamesNodes(context.node, context.name, path, envTypes)) do
				table.insert(result, {name = v.name, type = v.type or getPossibleType(v.node, envTypes, GEnvironmentTypes.All)})
			end
			
			return result
		end
	end
end


return module