Lexical Mode / Lexer Context


The lexical mode or lexer context is a lexer property while creating the token.

This is generally the only context related data.

It permits to apply lexer rules conditionally.

The most obvious example being the html pre or html code inside of which no other syntax should be recognized by the Lexer (ie no lexer rules should apply)

It means supporting different lexing rules depending on lexer context. It’s like having multiple separate sublexers:

  • one for each context.
  • that can be switched between.

This capability is necessary to tokenize some languages such as HTML 1)/XML 2)

Documentation / Reference

Powered by ComboStrap