4.5 lex_fn statement - specifying a custom lexer function

Propane generates both a lexer and a parser. By default, the parser uses the generated p_lex() function directly to return information for the next lexed token from the input stream.

However, the user can specify a custom lex function. This function may or may not use the Propane generated p_lex() function under the hood. For example, a token sequence could be injected or repeated from a previously saved macro definition.

Example (C/C++):

<<
static size_t mylexfn(p_context_t * context, p_token_info_t * out_token_info)
{
    static size_t count;
    size_t result = P_SUCCESS;
    if (count > 0)
    {
        out_token_info->token = TOKEN_a;
        out_token_info->pvalue = p_value(count);
        count--;
    }
    else
    {
        result = p_lex(context, out_token_info);
        if (out_token_info->token == TOKEN_c)
        {
            count = 3;
        }
    }
    return result;
}
>>

lex_fn mylexfn;

The lex_fn statement takes one argument specifying the name of the custom lexer function. The user must supply a value for the token field of the p_token_info_t output structure so that the parser knows what token was lexed.

Additionally, if the parser user code makes use of the token's pvalue, then the lexer function must supply a value for the pvalue field of the p_token_info_t structure. The p_value() generated API function could be useful for specifying parser values to associate with the lexed token when tree generation is not enabled. When tree generation is enabled, the pvalue field can be set to an instance of whatever type the user has defined as the ptype type.