4.3 drop statement - ignoring input patterns

A drop statement can be used to specify a lexer pattern that when matched should result in the matched input being dropped and lexing continuing after the matched input.

A common use for a drop statement would be to ignore whitespace sequences in the user input.

Example:

drop /\s+/;

4.4 free_token_node statement - freeing user-allocated memory in token node fields

If user lexer code block allocates memory to store in a token node's pvalue or any custom token user fields store pointers to allocated memory, the free_token_node grammar statement can be used to provide a code block which can be used to free memory properly.

Example freeing pvalue (C):

tree;
free_token_node <<
    free(${token.pvalue});
>>
ptype int *;
token a <<
  $$ = (int *)malloc(sizeof(int));
  *$$ = 1;
>>
token b <<
  $$ = (int *)malloc(sizeof(int));
  *$$ = 2;
>>
Start -> a:a b:b;

Example freeing custom token user fields (C):

token_user_fields <<
    char * comments;
>>
on_token_node <<
    ${token.comments} = (char *)malloc(some_len);
>>
free_token_node <<
    free(${token.comments});
>>

The free_token_node statement user code block is not emitted for D language since D has a garbage collector.