chickadee » glex

glex

Glex is a simple SRE-based lexer/tokenizer designed to be a good match for the acetone parser.

Glex is a macro. As arguments, you give it any amount of pairs of SRE and tags. The macro returns a procedure that can take a single rope, any amount of strings (they'll be concatenated), or nothing (in case it'll read from the current input port.)

((glex ("ab" ab) ("abb" abb)) "klabb in the ab")
⇒ ((unknown "kl") (abb "abb") (unknown " in the ") (ab "ab"))

You can use two-element lists, or pairs, it doesn't matter which:

((glex ("ab" . ab) ("abb" . abb)) "klabb in the ab")
⇒ ((unknown "kl") (abb "abb") (unknown " in the ") (ab "ab"))

You can use any SRE:

((glex (word word)) "klabb in the ab")
⇒ ((word "klabb")
   (unknown " ")
   (word "in")
   (unknown " ")
   (word "the")
   (unknown " ")
   (word "ab"))

You don't need to worry about PCRE grawlix mucking things up:

((glex ("^hi" literal-caret-hi)) "oh ^hi")

⇒ ((unknown "oh ") (literal-caret-hi "^hi"))

You can not use variables and stuff, it doesn't compose very well (your only recourse if you want some regexes containing other regexes is to construct the glex call itself from another macro). It's not a parser, it's just a tokenizer.

For a repo,

git clone https://idiomdrottning.org/glex