Delimiter ParseTreeNode missing

Oct 12, 2012 at 6:03 AM
Hi, the grammar
 
    public class TestGrammar : Grammar
    {
        public TestGrammar()
        {
            var items = new NonTerminal("items");
            var item = ToTerm("item");
            var delimiter = ToTerm("d");
            items.Rule = MakePlusRule(items, delimiter, item);
            this.Root = items;
        }
    }


 
with input "item d item" generates two item ParseTreeNode, but delimiter ParseTreeNode is missing, how to generate delimiter ParseTreeNode? Thanks.
Oct 12, 2012 at 2:20 PM

MakePlusRule automatically tags delimiter as punctuation, and omits it from the Parse tree. This behaviour seems to arise from the IsList flag being set on the non-terminal items. Why do you need other behaviour? The code below shows how I tested this.

   public TestGrammar() {
      var items = new NonTerminal("items");
//    items.Flags |= TermFlags.IsList;
      var item = ToTerm("item");
      var delimiter = ToTerm("d");
//    items.Rule = MakePlusRule(items, delimiter, item);
      items.Rule = item | items + delimiter + item;
      this.Root = items;
   }

Oct 12, 2012 at 2:56 PM

Hi pgeerkens, I'm using Irony to create a C# parser, and create Roslyn(http://msdn.microsoft.com/en-US/roslyn) syntax tree from the parse tree. In Roslyn, every terminal is important, I need delimiter ParseTreeNode. Your way works, but result list is not flat...


Oct 12, 2012 at 3:28 PM
Edited Oct 12, 2012 at 3:49 PM

Perhaps you can live [edit] with the the result of this grammar:

   public TestGrammar() {
      var items = new NonTerminal("items");
      var itemsTail = new NonTerminal("items");

      items.Flags |= TermFlags.IsList;
      var item = ToTerm("item");
      var delimiter = ToTerm("d");
      items.Rule = item | items + itemsTail;
      itemsTail.Rule	= delimiter + item;
      this.Root = items;
   }

Alternatively , you can dig into this routine in ReduceParserActions.cs, to achieve your desired effect:

  /// <summary>Reduces list created by MakePlusRule or MakeListRule methods. </summary>
  public class ReduceListBuilderParserAction : ReduceParserAction {
    
    public ReduceListBuilderParserAction(Production production) : base(production) { }

    protected override ParseTreeNode GetResultNode(ParsingContext context) {
      int childCount = Production.RValues.Count;
      int firstChildIndex = context.ParserStack.Count - childCount;
      var listNode = context.ParserStack[firstChildIndex]; //get the list already created - it is the first child node
      listNode.Span = context.ComputeStackRangeSpan(childCount);
      var listMember = context.ParserStack.Top; //next list member is the last child - at the top of the stack
      if (listMember.IsPunctuationOrEmptyTransient())
        return listNode;
      listNode.ChildNodes.Add(listMember);
      return listNode; 
    }
  }//class

Oct 16, 2012 at 7:53 AM

Hi Roman, could you please add a flag to TermListOptions such as EmitDelimiterParseTreeNode?

Coordinator
Oct 16, 2012 at 5:34 PM

Can you please show what a Roslyn's parse tree looks like (for a list like 'enum' declaration):

public enum Numbers { one, two, three }

Oct 17, 2012 at 5:49 AM

Hi Roman, Here is what Roslyn syntax tree looks like. As you can see, in Roslyn, every thing(including whitespace and comment) is important.

Coordinator
Oct 17, 2012 at 6:00 AM

I see. Not sure if it's really "important" :) . Irony's default arrangement is to construct the tree of nodes that "matter". Commas and delimiters are just "sugar" helping parser to separate stuff. Several years and many devs working with Irony - and yours is the first request for this. Before now everybody was happy with lists handling in Irony, so I'm kinda reluctant to add support for this into the parser. 

As a hack, try hooking to Reduced events of the lists and see if you can recover the token and add it back to the list before the last node. 

Roman