Thanks for the feedback Roman, but it seems like I was on the right track; the solution above works correctly, and the problem I encountered was being caused by a problem with the scanner. Although it does require the custom hint, I'm pretty satisfied
with the result, so unless I find a real issue with it, I'll go with my own solution.
Regarding the bug I think I've found, I have two kinds of identifiers allowed in my grammar: regular identifiers (which are purely alphabetic), and "built in" variables, which must begin with a %% prefix. I've defined these as follows:
var identifier = new IdentifierTerminal("identifier");
var builtin_variable = new IdentifierTerminal("builtin_variable");
This works fine normally (i.e. expressions are correctly parsed with references to identifier or builtin_variable as appropriate), but it seems that Scanner.GetToken() doesn't work correctly with the prefixed tokens when running in preview mode?
If I debug my custom action handler code above, the first call to scanner.GetToken() returns different results depending on whether the call to scanner.BeginPreview() is executed or not. If preview mode is inactive, it correctly returns an identifier token;
when active, it returns a builtin_variable token.
This behaviour is also apparent if I remove my custom action handler from the grammar altogether, in which case the identifiers are parsed correctly - as soon as I add my handler in, it incorrectly matches identifiers using the builtin_variable rule, which
causes reduction problems.
This seems to be a bug, but I can't see any obvious reason from a quick inspection of the scanner code why this might be happening?
PS: I also thought that if I used scanner.EndPreview(false), it might discard the incorrectly scanned tokens, and then correctly re-scan them when it continued after the handler, but instead it appears to completely discard them from the input queue? Is there
any way to force the scanner to re-scan them?