4.3 drop statement - ignoring input patterns

A drop statement can be used to specify a lexer pattern that when matched should result in the matched input being dropped and lexing continuing after the matched input.

A common use for a drop statement would be to ignore whitespace sequences in the user input.

Example:

drop /\s+/;

4.4 free_token_node statement - freeing user-allocated memory in token node fields

If user lexer code block allocates memory to store in a token node's pvalue or any custom token user fields store pointers to allocated memory, the free_token_node grammar statement can be used to provide a code block which can be used to free memory properly.

Example freeing pvalue (C):

tree;
free_token_node <<
    free(${token.pvalue});
>>
ptype int *;
token a <<
  $$ = (int *)malloc(sizeof(int));
  *$$ = 1;
>>
token b <<
  $$ = (int *)malloc(sizeof(int));
  *$$ = 2;
>>
Start -> a:a b:b;

Example freeing custom token user fields (C):

token_user_fields <<
    char * comments;
>>
on_token_node <<
    ${token.comments} = (char *)malloc(some_len);
>>
free_token_node <<
    free(${token.comments});
>>

Example freeing pvalue (Rust):

tree;
free_token_node <<
    if !${token.pvalue}.is_null()
    {
        unsafe { drop(Box::from_raw(${token.pvalue})); }
    }
>>
ptype *mut i32;
token a <<
  $$ = Box::into_raw(Box::new(1));
>>
token b <<
  $$ = Box::into_raw(Box::new(2));
>>
Start -> a:a b:b;

The free_token_node statement user code block is not emitted for D language since D has a garbage collector.

The code block is emitted for the Rust target, where it is run from a Drop implementation generated for p_context_t. It therefore runs exactly once however the context is disposed of, whether that is by calling p_context_delete() or by simply letting the context go out of scope. A ptype or token user field which owns its memory (a String, a Vec, a Box, and so on) is released when the context is dropped and does not need a free_token_node code block. The statement is only needed for memory which Rust does not track, such as a raw pointer obtained from Box::into_raw().