Antlr - Token

Card Puncher Data Processing




A token can be defined via:

Lexer rule

A token is primarily defined via a lexer rule (Lexical rule)


LOWERCASE = [a-z]+

Token Definition Section

The Token definition section of the grammar is a section for token with no associated lexical rule. The tokens section defines a set of tokens to add to the overall set.

The basic syntax is:

tokens { Token1, ..., TokenN }

Usage: Most of the time, the tokens section is used to define token types needed by actions in the grammar

Example: explicitly define keyword token types to avoid implicit definition warnings

tokens { BEGIN, END, IF, THEN, WHILE }

@lexer::members { // keywords map used in lexer to assign token types
Map<String,Integer> keywords = new HashMap<String,Integer>() {{
    put("begin", KeywordsParser.BEGIN);
    put("end", KeywordsParser.END);


token have their own file after generation

grammar Tok;
tokens { A, B, C }
a : X ;
  * [[generation|generation the lexer/parser...]]
<code bash>
antlr4 Tok.g4
warning(125): Tok.g4:3:4: implicit definition of token X in parser

  • A tokens file was created
cat Tok.tokens

Documentation / Reference

Discover More
Card Puncher Data Processing
Antlr - (Grammar|Lexicon) (g4)

Grammar in the context of Antlr. The grammar definition of Antlr is called a antlr/antlr4/blob/master/doc/lexicon.mdLexicon because the grammar is used by the lexer (hence the lexer grammar) See: ...
Card Puncher Data Processing
Antlr - (Lexical) Rule

in Antlr. Antlr has two types of rule: Name Case Type Description Example from the getting started uppercase letter lexer rule (known as Token name, they defines the token that the lexer...
Card Puncher Data Processing
Antlr - Generated class

From the grammar The lexer rules will create the lexer class The parser rules will create the parser class The classes generated will contain a method for each rule in the grammar. See from...
Idea Antlr Right Click Options
Antlr - Getting Started (Hello World)

A getting started page that brings you in the world of Antlr. antlr/antlr4/blob/master/doc/getting-started.mdantlr4 getting-started Create a grammar file called Hello.g4 and define the grammar...
Card Puncher Data Processing
Antlr - Lexer Rule (Token names|Lexical Rule)

in Antlr. They are rules that defines tokens. They are written generally in the grammar but may be written in a lexer grammar file Each lexer rule is either matched or not so every lexer rule expression...
Card Puncher Data Processing
Antlr - Lexical mode

Lexical mode in Antlr Lexical modes allow to split a single lexer grammar file into multiple sublexers. The lexer can then only return tokens matched by rules from the current mode. Lexers start out...
Card Puncher Data Processing
Antlr - Parser Rule

in Antlr. Parser rule is the second type of rule for Antlr. They begin with a lowercase letter. The lexer rules specify the tokens whereas the parser rules specify the tree. URL URI See ...

Share this page:
Follow us:
Task Runner