Lexer rule

A token is primarily defined via a lexer rule (Lexical rule)

Example:

• the lexical rule LOWERCASE that capture a string of lowercase characters.

LOWERCASE = [a-z]+

Token Definition Section

The Token definition section of the grammar is a section for token with no associated lexical rule. The tokens section defines a set of tokens to add to the overall set.

The basic syntax is:

tokens { Token1, ..., TokenN }

Usage: Most of the time, the tokens section is used to define token types needed by actions in the grammar

Example: explicitly define keyword token types to avoid implicit definition warnings

tokens { BEGIN, END, IF, THEN, WHILE }

@lexer::members { // keywords map used in lexer to assign token types
Map<String,Integer> keywords = new HashMap<String,Integer>() {{
    put("begin", KeywordsParser.BEGIN);
    put("end", KeywordsParser.END);
    ...
}};
}