Resolving errors by injecting tokens - demo

Mar 22, 2014 at 9:24 PM
Edited Mar 22, 2014 at 9:27 PM
I need XML highlighting for one of my projects. Problem is that recovering from errors by skipping to the end of the line (used in irony samples) is not very usable for XML. So i modified irony a little to support recovering from errors by injecting tokens. I think it can be useful for others, and maybe used as baseline for implementing some official support.

First of all link to the modified version of Irony package.
modified files are:
  • Grammar.cs - new virtual method TryInjectRecoveryTokens called from ErrorRecoveryParserAction
  • ErrorRecoveryParserAction.cs - modified Execute to call TryInjectRecoveryTokens on grammar
  • ParserStack.cs - modified Pop(int) & PopUntil(int) to actually return popped items (required for propagating injected status of ParseTreeNodes
  • ReduceParserActions.cs - added the propagation of injected flag to the result of this action
  • ParseTree.cs - Added IsInjected flag to the ParseTreeNode and HasInjections() method to the ParseTree
  • Parser.cs - changed access modifier of GetNextAction to public
  • ParsingContext.cs - changed access modifier of ParserStack, CurrentParserState, CurrentParserInput to public
  • SourceStream. - Added support to SetNewPosition to index smaller than actual position. (addition optimization could be done)
  • Token.cs - added IsInjected TokenFlag and IsInjected() method
  • fm.GrammerExplorer.cs - added green text color to Parse Tree nodes which contains injected tokens.
The grammar with injection handling code link

The grammar returns only linear list of tags and content. (I am handling the xml structure separately)
Mar 24, 2014 at 5:25 PM
thanks, will look at this. I'm working (unfortunately really sloooowly) on new version, and token injection is one of new things I plan to add.
Mar 24, 2014 at 6:52 PM
Edited Mar 24, 2014 at 6:55 PM
The main problem i see in my implementation is rescanning the last token returned from scanner after the injection. There should be better way than returning to start of the line of the last token and simply recalculating the columns and tabs. Probably something like caching positions of last X tokens inside the scanner and method to rollback one or more tokens. Or maybe just not moving Position to PreviewPosition before the error is handled, but i don't know the internals of the preview mechanism enough to do it.

I don't think it's possible to add something like ErrorRule to do the injections. Because the injection is adding terminal tokens between the ParseStack.Top and CurrentToken and there are two possible situations:
  1. Something is missing to complete current NonTerminal -> i have to inject some tokens and rescan current token. And this can actually depend on more than one item on ParserStack.
  2. Current token is obstruction for parser so i have to inject something before it to make it reducible to nonterminal.
And i have few additional questions
  • Is there something like MakeQuestionmarkRule() ? i don't like the "| Empty" i am currently using.
  • Is there a way to identify term with plus in Name differently than comparing its Name? Because if i change the Name Parameter of Attribute to something else than "Attribute" my code would stop working..