Antlr - Token



A token can be defined via:

Lexer rule

A token is primarily defined via a lexer rule (Lexical rule)


LOWERCASE = [a-z]+

Token Definition Section

The Token definition section of the grammar is a section for token with no associated lexical rule. The tokens section defines a set of tokens to add to the overall set.

The basic syntax is:

tokens { Token1, ..., TokenN }

Usage: Most of the time, the tokens section is used to define token types needed by actions in the grammar

Example: explicitly define keyword token types to avoid implicit definition warnings

tokens { BEGIN, END, IF, THEN, WHILE }

@lexer::members { // keywords map used in lexer to assign token types
Map<String,Integer> keywords = new HashMap<String,Integer>() {{
    put("begin", KeywordsParser.BEGIN);
    put("end", KeywordsParser.END);


token have their own file after generation

grammar Tok;
tokens { A, B, C }
a : X ;
  * [[generation|generation the lexer/parser...]]
<code bash>
antlr4 Tok.g4
warning(125): Tok.g4:3:4: implicit definition of token X in parser
  • A tokens file was created
cat Tok.tokens

Documentation / Reference

Powered by ComboStrap