Creating AstNode derived classes

Jun 26, 2012 at 6:28 PM

Hi - I'm new to Irony and was trying to build a simple 4-function calculator grammar to see how things work. Defining the grammar and parsing a test string as input seems easy enough, but I'm a bit lost when trying to add code for processing the nodes in the parse tree. Some posts say that I should define a class derived from AstNode for each type of node in the grammar, but the AstNode class doesn't seem to exist in the latest version of Irony.dll. Is there a new way of doing this? Are there any simple example projects that show how this is done?

Many thanks.

Coordinator
Jun 26, 2012 at 6:52 PM

man... ExpressionEvaluator in samples - did you see it?

Jun 26, 2012 at 7:17 PM

Yes I looked at the ExpressionEvaluator, but looking at it again, I've discovered another DLL, Irony.Interpreter.dll, that I hadn't found earlier, and it seems to have AstNode in it, so will give it another try. Thanks for your reply.

Jun 26, 2012 at 8:21 PM

Sorry, but I'm still unable to get it to work. I've defined the grammar as follows:

 

  public class TestGrammar : Grammar
  {
    public TestGrammar() : base(false)  // Set CaseSensitive to false in base class
    {
      var number = new NumberLiteral("number");
      var BinOp = new NonTerminal("BinOp");
      var Statement = new NonTerminal("Statement");

      Statement.Rule = number + BinOp + number;
      BinOp.Rule = ToTerm("+") | "-";
      Root = Statement;
      LanguageFlags = LanguageFlags.NewLineBeforeEOF | LanguageFlags.CreateAst;
    }
  }

I use the grammar in Main as follows:
  class Program
  {
    static void Main(string[] args)
    {
      TestGrammar grammar = new TestGrammar();
      LanguageData language = new LanguageData(grammar);
      LanguageRuntime runtime = new LanguageRuntime(language);
      Parser parser = new Parser(language);
      ParseTree parseTree = parser.Parse("3+4");
      ParseTreeNode root = parseTree.Root;
      dispTree(root, 0);
    }

     public static void dispTree(ParseTreeNode node, int level)
    {
      for (int i = 0; i < level; i++)
        Console.Write("  ");
      Console.WriteLine(node);

      foreach (ParseTreeNode child in node.ChildNodes)
        dispTree(child, level + 1);
    }
 }

However, the program hits a NullReferenceException in AstBuilder.BuildAst when attempting to process a 'number' node, 
at the line shown:

      if (config.NodeCreator != null) {
        config.NodeCreator(Context, parseNode);
        // We assume that Node creator method creates node and initializes it, so parser does not need to call 
        // IAstNodeInit.Init() method on node object. But we do call AstNodeCreated custom event on term.
      } else {
        //Invoke the default creator compiled when we verified the data
        parseNode.AstNode = config.DefaultNodeCreator(); <------ Exception at this line

The problem appears to be that neither NodeCreator nor DefaultNodeCreator has been set.
Thanks again.

 

Coordinator
Jun 26, 2012 at 8:26 PM

Look at ExpressionEvaluatorGrammar - you have to explicitly provide AstNode type to constructors of Terminals and non-terminals

Jun 26, 2012 at 8:39 PM

The exception occurs on processing a node of type NumberLiteral, not a Terminal or NonTerminal. In ExpressionEvaluatorGrammar, 'number' isn't declared with an explicit type. The code in ExpressionEvaluatorGrammar where 'number' is declared is

      // 1. Terminals
      var number = new NumberLiteral("number");
      //Let's allow big integers (with unlimited number of digits):
      number.DefaultIntTypes = new TypeCode[] { TypeCode.Int32, TypeCode.Int64, NumberLiteral.TypeCodeBigInt };

I tried adding the line below to my own code...

      number.DefaultIntTypes = new TypeCode[] { TypeCode.Int32, TypeCode.Int64, NumberLiteral.TypeCodeBigInt };

However, the exception still occurs.

Coordinator
Jun 26, 2012 at 8:41 PM

NumberLiteral IS Terminal. Just copy all from expression evaluator grammar and start changing it to what you need.

Roman

Dec 30, 2012 at 3:49 AM

Having same problem.

Not to be too harsh, but a NumberLiteral might be a Terminal -- but in the Expression Evaluator sample there is no AstNode being set on it. 
The above lines are identical to what is in the sample. 

Where was growe supposed to define the typed ASTNode for the Terminal token?

Thanks.

 

 

 

 

 

Oct 9, 2013 at 1:54 PM
Edited Oct 9, 2013 at 2:22 PM
I am having the exact same problem. Unfortunately, the Expression Evaluator sample is of little help here.

I finally found a solution by applying the following code:
var number = TerminalFactory.CreateCSharpNumber("number");
number.AstConfig.NodeType = typeof (MyNumber);
Where MyNumber inherits IAstNodeInit, however, it's not very straightforward.

One approach to make it more intuitive would be to extend the method CreateCSharpNumber with an overload taking a System.Type as a second argument.

Joannes
Oct 9, 2013 at 9:01 PM
I dealt with things like that in my Shakespeare compiler:

Article:
http://honestillusion.com/blog/2013/09/12/Shakespeare-compiler-for-NET/

Source Code:
https://github.com/jamescurran/ShakespeareCompiler
Coordinator
Oct 10, 2013 at 5:59 PM
for standard terminals like Number, there's a default LiteralNode AST node that interpreter uses. But you can set your own explicitly.
Mar 27, 2014 at 6:40 PM
rivantsov wrote:
NumberLiteral IS Terminal. Just copy all from expression evaluator grammar and start changing it to what you need. Roman
I'm having problems with this also. I've taken your advice and used the ExpressionEvaluatorGrammar sample as the starting point, but that doesn't seem to help. All the code below is based on the BETA from Tue Sep 24, 2013.

I've removed everything from the ExpressionEvaluatorGrammar to give just this:
using Irony.Interpreter;
using Irony.Interpreter.Ast;
using Irony.Parsing;

namespace FreightLanguageCompiler
{
    [Language("SK-ExpressionEvaluator", "1.0", "Multi-line expression evaluator")]
    public class SKExpressionEvaluatorGrammar : InterpretedLanguageGrammar
    {
        public SKExpressionEvaluatorGrammar()
            : base(caseSensitive: false)
        {
            var number = new NumberLiteral("number");
            var Statement = new NonTerminal("Statement");
            var Program = new NonTerminal("Program", typeof (StatementListNode));

            Statement.Rule = number;
            Program.Rule = MakePlusRule(Program, NewLine, Statement);

            Root = Program; 
            MarkTransient(Statement);
            LanguageFlags = LanguageFlags.NewLineBeforeEOF | LanguageFlags.CreateAst | LanguageFlags.SupportsBigInt;
        }
    }
}
Parsing anything then gives an NRE in AstBuilder in the line shown below as DefaultNodeCreator is not assigned when called:
    public virtual void BuildAst(ParseTreeNode parseNode) {
      var term = parseNode.Term;
      if (term.Flags.IsSet(TermFlags.NoAstNode) || parseNode.AstNode != null) return; 
      //children first
      var processChildren = !parseNode.Term.Flags.IsSet(TermFlags.AstDelayChildren) && parseNode.ChildNodes.Count > 0;
      if (processChildren) {
        var mappedChildNodes = parseNode.GetMappedChildNodes();
        for (int i = 0; i < mappedChildNodes.Count; i++)
          BuildAst(mappedChildNodes[i]);
      }
      //create the node
      //We know that either NodeCreator or DefaultNodeCreator is set; VerifyAstData create the DefaultNodeCreator
      var config = term.AstConfig;
      if (config.NodeCreator != null) {
        config.NodeCreator(Context, parseNode);
        // We assume that Node creator method creates node and initializes it, so parser does not need to call 
        // IAstNodeInit.Init() method on node object. But we do call AstNodeCreated custom event on term.
      } else {
        //Invoke the default creator compiled when we verified the data
        parseNode.AstNode = config.DefaultNodeCreator();  <<<=== NRE HERE
        //Initialize node
        var iInit = parseNode.AstNode as IAstNodeInit;
        if (iInit != null)
          iInit.Init(Context, parseNode);
      }
      //Invoke the event on term
      term.OnAstNodeCreated(parseNode);
    }//method
I've tried various combinations of advice in this thread but no avail. If I can't fix this then I'm sadly going to have to switch from Irony, which I would really prefer not to have to do! :(
Coordinator
Mar 27, 2014 at 7:08 PM
believe it or not, can't reproduce it. Just downloaded Irony zip from Download page, unzipped in temp folder, copied your grammar, ran it with '15' as an input, and it works fine
Did you already 'hack' something inside Irony?
Mar 27, 2014 at 7:27 PM
Okay, I've zipped the solution and put it here:

http://1drv.ms/1foo2d8

The class is SKExpressionEvaluatorGrammar. Let me know if that reproduces the behaviour.
Coordinator
Mar 27, 2014 at 7:46 PM
weeeellll, got it. In your listing of grammar above, it is inherited from InterpretedLanguageGrammar. In your zip, it derives from Grammar. And this makes all the difference.
Next, it does give you an error message when you try to parse:
_AstNodeType or AstNodeCreator is not set on non-terminals: number. Either set Term.AstConfig.NodeType, or provide default values in AstContext._
Look in InterpretedLanguageGrammar, BuildAst method, it properly sets up the AstContext (instance of interpreterAstContext) - which has this default ast node setup properly.
So either use InterpretedLanguageGrammar as a base, or borrow some code from it to setup things right. Or provide AstNodeType explicitly for everything.
I do agree that this all is quite non-intuitive, will try to fix in the next version - I'm working on big overhaul of the entire thing.
Sorry for trouble, and thanks for the favorable review.
Roman
Mar 27, 2014 at 7:54 PM
Yesssss, thank you! :) That does indeed sort it.

I'd agree that this part has been a bit unintuitive, but the Irony use of syntax to build the grammar is just a joy to use and really easy to pick up. Can't wait to see the next version.

I really appreciate the help and the swift response too. :)