Lexer are also known as:
- Lexical analyzer
- Lexical Tokenizer
- Lexical Scanner
Consider the following expression:
sum = 3 + 2;
A lexer will tokenized it in the following symbol table:
|;||End of statement|
A lexer defines how the contents of a file is broken into tokens.
A lexer reads an input character or byte stream (i.e. characters, binary data, etc.), divides it into tokens using:
- and generates a token stream as output.
- lexical analysis,
- or simply scanning.
A Lexer is a stateful stream generator (ie the position in the source file is saved). Every time it is advanced, it returns the next token in the Source. Normally, the final Token emitted by the lexer is a EOF and it will repeatedly return the same EOF token whenever called.
Lexers are generally quite simple and does nothing with the tokens. Most of the complexity is deferred to the following steps:
- or semantic analysis 1) (seeing if there is a problem, a possible optimization, …)
For example, a typical lexical analyzer recognizes parentheses as tokens, but does nothing to ensure that each “(” is matched with a “)”. This syntax analysis is left to the parser.
Tokenizing is generally done in a single pass.
- Categorize each lexeme (sequence of characters) into tokens (symbols of the vocabulary of the language). If the lexer finds an invalid token, it will report an error.
Lexers can be generated by automated tools called compiler-compiler.
They are working:
- from a grammar file that defines the lexer rules
- then generates the lexer as code
You can also write your own lexer manually.
They all use ultimately a finite automaton modeling the recognition of the keyword then
- the state (circle) represents a different position in the word then that has been reached so far that ranges from:
- the empty string (ie nothing)
- to the complete word
Why not using simply a regular expression (Lexer vs Regexp)
Why not using simply a regular expression