Lexical Mode / Lexer Context

Compiler

About

The lexical mode is a lexer property while creating the token.

It's also known as:

  • lexer context
  • or lexer state

This is generally the only context related data.

It permits to apply lexer rules conditionally.

It means supporting different lexing rules depending on the lexer context. It’s like having multiple separate sub-lexers:

  • one for each context.
  • that can be switched between.

Example

The most obvious example is the html pre or html code inside of which no other syntax should be recognized by the Lexer (ie no lexer rules should apply)

This capability is also necessary to tokenize some languages such as HTML 1)/XML 2)

Name in Library

Chevrotain

push_mode in chevrotain

Antlr

mode, push_mode in antlr

Jflex

Mode is called a state in Jflex.

You enter a state with the function yybegin. Jflex starts with the state called YYINITIAL

yybegin(MY_STATE)

Jflex does not have the notion of push mode, you need to implement it yourself, Example

%s[tate] "state identifier" [, "state identifier", ... ] for inclusive or
%x[state] "state identifier" [, "state identifier", ... ] for exclusive states

3) 4) 5)





Discover More
Card Puncher Data Processing
Antlr - Lexical mode

Lexical mode in Antlr Lexical modes allow to split a single lexer grammar file into multiple sublexers. The lexer can then only return tokens matched by rules from the current mode. Lexers start out...
Compiler
Compiler - Lexer rule (Token names, Lexical rule, Token name)

lexer rule are rules written specifically for the lexer as they defines tokens The lexer creates token from the input text that match this rules. A lexer rule is also known as: lexical rule token...
Compiler
Language - Context Free Grammar (CFG)

A context-free grammar is: a set of instructions called grammar rules (known as production rules or production) that describe all possible strings in a given formal language without context ...
Word Recognition Automaton
What is a Lexer ? known also as Tokenizer or Scanner - Lexical Analysis

Lexer are also known as: * Lexical analyzer * Lexical Tokenizer * Lexical Scanner Consider the following expression: A lexer will tokenized it in the following symbol table: Token Lexeme ...



Share this page:
Follow us:
Task Runner