4.12 token_user_fields statement - adding custom token fields

When tree generation mode is active, Propane generates a tree node structure and a token node structure for each matching rule and token instance in the input string. The user may add custom fields to token tree nodes using the token_user_fields statement. The code block supplied to the token_user_fields is inserted in the struct generated by the parser to hold a token tree node.

Example (D/C++):

token_user_fields <<
    string mytokenval;
>>

The on_token_node statement can be used to provide code that initializes any token user fields when a token tree node instance is created.

For example (C++):

context_user_fields <<
    std::string comments;
>>
token_user_fields <<
    std::string comments;
>>
on_token_node <<
    ${token.comments} = ${context.comments};
    ${context.comments} = "";
>>
drop /#(.*)\n/ <<
    /* Accumulate comments before the next parser tree node. */
    ${context.comments} += std::string((const char *)match, match_length);
>>

If a pointer to any allocated memory is stored in a user-defined context field, the free_token_node statement can be used to supply a code block which will be executed immediately before the token node is freed. For C++, the delete statement is used to free the token tree node, so the destructor for any custom token user fields will be called.