The token statement allows defining a lexer token and a pattern to match that token. The name of the token must be specified immediately following the token keyword. A regular expression pattern may optionally follow the token name. If a regular expression pattern is not specified, the name of the token is taken to be the pattern. See also: Regular expression syntax.

Example:

token for;

In this example, the token name is for and the pattern to match it is /for/.

Example:

token lbrace /\{/;

In this example, the token name is lbrace and a single left curly brace will match it.

The token statement can also include a user code block. The user code block will be executed whenever the token is matched by the lexer.

Example:

token if <<
  writeln("'if' keyword lexed");
>>

The token statement is actually a shortcut statement for a combination of a tokenid statement and a pattern statement. To define a lexer token without an associated pattern to match it, use a tokenid statement. To define a lexer pattern that may or may not result in a matched token, use a pattern statement.