4.5 lex_fn statement - specifying a custom lexer function

Propane generates both a lexer and a parser. By default, the parser uses the generated p_lex() function directly to return information for the next lexed token from the input stream.

However, the user can specify a custom lex function. This function may or may not use the Propane generated p_lex() function under the hood. For example, a token sequence could be injected or repeated from a previously saved macro definition.

Example (C/C++):

<<
static size_t mylexfn(p_context_t * context, p_token_info_t * out_token_info)
{
    static size_t count;
    size_t result = P_SUCCESS;
    if (count > 0)
    {
        out_token_info->token = TOKEN_a;
        out_token_info->pvalue = p_value(count);
        count--;
    }
    else
    {
        result = p_lex(context, out_token_info);
        if (out_token_info->token == TOKEN_c)
        {
            count = 3;
        }
    }
    return result;
}
>>

lex_fn mylexfn;

The lex_fn statement takes one argument specifying the name of the custom lexer function. The user must supply a value for the token field of the p_token_info_t output structure so that the parser knows what token was lexed.

Additionally, if the parser user code makes use of the token's pvalue, then the lexer function must supply a value for the pvalue field of the p_token_info_t structure. The p_value() generated API function could be useful for specifying parser values to associate with the lexed token when tree generation is not enabled. When tree generation is enabled, the pvalue field can be set to an instance of whatever type the user has defined as the ptype type.

The custom lex function returns a result code to the parser. Returning P_SUCCESS indicates that the function has produced a token in the token output field and the parse should proceed. Returning any other result code stops the parse immediately; the parser propagates that code out of p_parse() (and the p_parse_XXX() and p_parse_inner_XXX() functions) unchanged. This allows a custom lex function to surface an error condition, for example by propagating a P_UNEXPECTED_TOKEN, P_UNEXPECTED_INPUT, or P_DECODE_ERROR result from a nested p_parse_inner_XXX() or p_lex() call.

Observe the following contract when returning a result code other than P_SUCCESS:

• Do not return P_DROP. This code is an internal lexer signal and is never returned to the parser by the generated p_lex() function. Returning it from a custom lex function would be reported as a spurious parse failure. If drop functionality is required, the custom lex function should loop and return the next non-drop token.
• Do not return P_EOF to signal the end of the input. The end of input is communicated to the parser by returning P_SUCCESS with the token field set to the end-of-input token (TOKEN___EOF), which is what the generated p_lex() function does. Returning P_EOF aborts the parse rather than allowing it to complete.
• The p_token() and p_position() accessors are populated by the parser only when the parser itself detects an unexpected token. When a custom lex function returns an error code, p_token() is not updated and may not reflect a meaningful token, and p_position() reflects the lexer's current text position rather than a parser-identified error location. A custom lex function that wants a specific reported position can set it with p_set_position() before returning.
• Returning P_USER_TERMINATED does not populate the user terminate code returned by p_user_terminate_code(). This user terminate code is normally populated by the $terminate() function in the user code block.