Parser Creation

Doomcolp · January 10, 2024, 8:53pm

I know I’ve already asked a similar question and got a solution in this post but this was at the time when I wanted to create my own little custom language and I kinda lost hope for that / gave up. There’s good resources that were linked but I can’t seem to turn the info from those pages into actual Lua code for the custom language so I doubt I’d be able to create a Lua parser with that.

Now I want to create a Lua(U) parser (that generates an AST) but there’s only really this post. It’s good but the parser is for a Math Evaluator which is not really what I’m trying to do.

Well I know I started the last paragraph off with “Now I want to” but really I’ve been wanted to do this months before I wanted to create a custom language. Don’t know why I tried posting that before this but this is the important one I guess.

There’s little resources on the devforum about creating your own parser. So if this gets a good solution than other people in the future will have a good starting point.

Thank you!

Are you having trouble creating a parser (specifically a Lua one)?

Yes
No
I’m not tryna make one

0 voters

azqjanna · January 11, 2024, 12:13am

I would look at the documentation for Flex / Bison, which are parser generator tools for C. Reading them will give you a better idea of what is required and what they are able to do. I would recommend making an LR parser and then a LALR(1) parser to get the full explanation. Both are stack machines but their rules are very easy to write incorrectly. I have one written in Lua but I forget which kind it is:
parse.lua (9.3 KB)

Doomcolp · January 11, 2024, 6:53am

Sorry for the real late response. I did like your post a couple hours ago but I didn’t really have the chance to reply.

I did check out the sources you recommended me to check out; both Flex and Bison. I’m not sure if Flex is exclusively Lexers but if not I can check again.

For Bison, I read a bit and I think it’s really interesting but I couldn’t really find anything about generating ASTs.

Since you’ve written a parser (possibly that generates ASTs) if there a chance you could try to explain the process? Or if Bison does have some sort of docs on generating ASTs then you could lmk.

By the way, thank you for everything you’ve already provided. It’s late for me atm so I kinda skimmed thru your parser but good job!

Sorry if I don’t get back to you for a bit cuz I got school tmmr

azqjanna · January 11, 2024, 12:19pm

Flex is a lexer but its designed to be built with a Bison parser at the same time. Bison generates parsers which generate ASTs.

Doomcolp · January 11, 2024, 7:57pm

Hm yeah that makes sense.

Ah I didn’t really see anything on the site that stated that. I mean it’s pretty obvious but I just wanted to confirm.

Doomcolp3 · January 15, 2024, 6:27am

I just reread over the parts in the Bison documentation about how the parser works and it seems like their parsers are really good but I’m not really getting the information I’m scouting for. It’s also kinda late and I’m in my room with no lights on. So maybe I’m still not catching something but oh well.

I’m looking for how I’d create a parser that can generate an AST or Abstract Syntax Tree.

Really just how they would be implemented in Luau or Lua code. Nothing too crazy.

(Yes I am on an alt account cuz I’m too lazy to sign back into my main account)

AstrealDev · January 16, 2024, 1:56pm

Hey I wrote one of the posts you linked. I have a parser for a language that is open-source and written in Luau from this project here. You can find all the code on the GitHub and I go over some of the details on the documentation website here.

Doomcolp · January 23, 2024, 8:14pm

I actually do not know how I liked your post and still managed to forget to reply for 7 days.

Thank you for making that post and making Toned but I’m still a bit in the dark lol.

(sorry for the necrobump)

PhoenixRessusection · January 23, 2024, 8:38pm

He basically meant learn from the source.

github.com

astrealRBLX/Toned/blob/master/src/ToneLang/Lexer.lua

local Lexer = {}
local LexerClass = {}

local ToneLang = script.Parent

local Types = require(ToneLang.Types)
local TokenModule = require(ToneLang.Token)

local Tokens = TokenModule.Tokens
local Token = TokenModule.Token

local Keywords = {
  ['true'] = Tokens.TRUE,
  ['false'] = Tokens.FALSE,
  ['var'] = Tokens.VAR,
  ['enum'] = Tokens.ENUM,
}

local WHITESPACE = {'', ' ', '\n', '\t', '\r', '\v', '\f'}
local function isWhitespace(char: string)

This file has been truncated. show original

github.com

astrealRBLX/Toned/blob/master/src/ToneLang/Parser.lua

local Parser = {}
local ParserClass = {}

local ToneLang = script.Parent

local Types = require(ToneLang.Types)
local Tokens = require(ToneLang.Token).Tokens
local Expr = require(ToneLang.Nodes.Expr)
local Stmt = require(ToneLang.Nodes.Stmt)
local Select = require(ToneLang.Nodes.Select)
local MiscNode = require(ToneLang.Nodes.MiscNode)

function Parser.new(tokens: { Types.Token })
  return setmetatable({
    position = 1,
    tokens = tokens,
  }, {
    __index = ParserClass
  })
end

This file has been truncated. show original

Doomcolp · January 23, 2024, 10:02pm

Shoot my bad I checked GitHub but missed that somehow. Thanks for pointing that out

Doomcolp · January 29, 2024, 2:31am

Sorry for marking the solution so late. Thank you AstroDev for sharing your resource! It’s definitely helped me understand the process of parsing more!

And thank you PheonixRessusection for pointing me to the source code!

system · February 12, 2024, 2:31am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.