Code colorizer error

Sep 5, 2013 at 3:51 AM
Edited Sep 5, 2013 at 3:53 AM
I am trying to write a grammar to parse a statement like
b.a := 7;
My grammar is as shown below
    public class STGrammar : InterpretedLanguageGrammar
    {
        public STGrammar() : base(caseSensitive: false)
        {
            LanguageFlags = LanguageFlags.CreateAst;

            # region Terminals
            Terminal n = new NumberLiteral("number");
            Terminal v = new IdentifierTerminal("variable");
            KeyTerm assignment = ToTerm(":=");
            KeyTerm left_paren = ToTerm("(");
            KeyTerm right_paren = ToTerm(")");
            KeyTerm terminate = ToTerm(";");
            KeyTerm left_sq = ToTerm("[");
            KeyTerm right_sq = ToTerm("]");
            KeyTerm dot = ToTerm(".");
            #endregion Terminals

            #region Nonterminals
            NonTerminal actionBlock = new NonTerminal("actionBlock", typeof(ActionBlock));
            NonTerminal assignmentExpr = new NonTerminal("assignmentExpr", typeof(AssignmentStNode));
            NonTerminal arrayOrStruct = new NonTerminal("arrayOrStruct", typeof(ArrayNode));
            NonTerminal arrayOrStructRHS = new NonTerminal("arrayOrStructRHS", typeof(AstNode));
            NonTerminal arrayOrStructRHSs = new NonTerminal("arrayOrStructRHSs");
            NonTerminal comparatorExp = new NonTerminal("comparatorExpression", typeof(ExpressionNode));
            #endregion Nonterminals

            #region BNFRules
            actionBlock.Rule =
                assignmentExpr + terminate
                | arrayOrStruct + terminate
                ;
            assignmentExpr.Rule =
                v + assignment + v
                | v + assignment + n
                | arrayOrStruct + assignment + v
                | v + assignment + arrayOrStruct
                | arrayOrStruct + assignment + arrayOrStruct
                | arrayOrStruct + assignment + n
                ;
            arrayOrStruct.Rule =
                v + arrayOrStructRHSs
                ;
            arrayOrStructRHS.Rule = 
                "." + v 
                | "[" + v + "]"
                ;
            arrayOrStructRHSs.Rule =
                MakePlusRule(arrayOrStructRHSs, arrayOrStructRHS)
                ;

            this.Root = actionBlock;
            
            #endregion BNFRules

            MarkPunctuation(",", ";", "[", "]", ")", "(", "..", ".");
            RegisterBracePair("(", ")");
        }
It works fine when I am trying to parse
b := 7;
The error coming is "Fatal Error in code colorizer. Colorizing had been disabled."
The detailed error is
System.NullReferenceException was unhandled by user code
  Message=Object reference not set to an instance of an object.
  Source=Irony
  StackTrace:
       at Irony.Ast.AstBuilder.BuildAst(ParseTreeNode parseNode) in C:\Sreeja\downloads\Irony_2013_03_10\Irony\Ast\AstBuilder.cs:line 97
       at Irony.Ast.AstBuilder.BuildAst(ParseTreeNode parseNode) in C:\Sreeja\downloads\Irony_2013_03_10\Irony\Ast\AstBuilder.cs:line 86
       at Irony.Ast.AstBuilder.BuildAst(ParseTreeNode parseNode) in C:\Sreeja\downloads\Irony_2013_03_10\Irony\Ast\AstBuilder.cs:line 86
       at Irony.Ast.AstBuilder.BuildAst(ParseTreeNode parseNode) in C:\Sreeja\downloads\Irony_2013_03_10\Irony\Ast\AstBuilder.cs:line 86
       at Irony.Ast.AstBuilder.BuildAst(ParseTree parseTree) in C:\Sreeja\downloads\Irony_2013_03_10\Irony\Ast\AstBuilder.cs:line 38
       at Irony.Interpreter.InterpretedLanguageGrammar.BuildAst(LanguageData language, ParseTree parseTree) in C:\Sreeja\downloads\Irony_2013_03_10\Irony.Interpreter\InterpretedLanguageGrammar.cs:line 60
       at Irony.Parsing.Parser.Parse(String sourceText, String fileName) in C:\Sreeja\downloads\Irony_2013_03_10\Irony\Parsing\Parser\Parser.cs:line 88
       at Irony.GrammarExplorer.fmGrammarExplorer.ParseSample() in C:\Sreeja\downloads\Irony_2013_03_10\Irony.GrammarExplorer\fmGrammarExplorer.cs:line 348
       at Irony.GrammarExplorer.fmGrammarExplorer.btnParse_Click(Object sender, EventArgs e) in C:\Sreeja\downloads\Irony_2013_03_10\Irony.GrammarExplorer\fmGrammarExplorer.cs:line 507
       at System.Windows.Forms.Control.OnClick(EventArgs e)
       at System.Windows.Forms.Button.OnClick(EventArgs e)
       at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
       at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
       at System.Windows.Forms.Control.WndProc(Message& m)
       at System.Windows.Forms.ButtonBase.WndProc(Message& m)
       at System.Windows.Forms.Button.WndProc(Message& m)
       at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
       at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
       at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
  InnerException: 
Can you help me to resolve the error?
I am using Irony_2013_03_10 version.

Sreeja
Coordinator
Sep 5, 2013 at 6:45 PM
Disable colorizing using checkbox on top in GrammarExplorer. Then paste the script line, hit Parse and use debugger to see what happens. Most likely you miss AstNodeType on some non-terminal. Colorizer just tries to do parsing in background, and it gets the same error you'd get when you hit Parse.
Sep 6, 2013 at 6:36 AM
Edited Sep 6, 2013 at 11:54 AM
The error being thrown from
parseNode.AstNode = config.DefaultNodeCreator();
It parses actionBlock, assignmentExpr, arrayOrStruct, arrayOrStructRHSs, arrayOrStructRHS. Till that it is fine. But I don't get why it is going again with arrayOrStructRHS, which again works fine but throwing exception in arrayOrStructRHSs which comes as the next step. I have only left the non terminal arrayOrStructRHSs which is a MakePlusRule of arrayOrStructRHS. Should I implement ASTNode for this also?

I have just made stub classes for AST nodes, like
public class ActionBlock : AstNode { }
public class AssignmentStNode : AstNode { }
public class ArrayNode : AstNode { }
public class ArrayRhsNode : AstNode { }
public class ExpressionNode : AstNode { }
Will that be a problem?

And while working with AST nodes, can you please tell me which all classes should I override apart from Init. What I understood is to override Init if we have to see the AST node. I saw Evaluate and SetEvaluate methods. I couldn't figure out the function of these. Should I implement any other method to make it working or the stub will do?

Thanks a lot for spending your valuable time for helping me. I am just a beginner to Irony and parsing as a whole.

Edit :

I implemented just the basic Init method for all the AST nodes to see whether they are working. Now I am getting an exception thrown and also an error in grammar explorer window as follows
AstNodeType or AstNodeCreator is not set on non-terminals: Irony.Parsing.BnfTermList. Either set Term.AstConfig.NodeType, or provide default values in AstContext.
The Init method which I implemented is
public override void Init(AstContext context, ParseTreeNode treeNode)
{
    base.Init(context, treeNode);
}
The parse tree is now being displayed correctly.

I tried overriding BuildAst() as per the discussion . But it is not working either.
Thanks,
Sreeja
Coordinator
Sep 9, 2013 at 9:20 AM
yes, you should add Ast node type for arrayOrStructRHSs nonterminal - all nonterminals must have Ast node type, except transient ones.
apart from Init - override DoEvaluate,
Sep 12, 2013 at 10:37 AM
I tried the suggestions you gave me and AST is being created. But in grammar explorer, in the parser output window there is an error
AstNodeType or AstNodeCreator is not set on non-terminals: Irony.Parsing.BnfTermList. Either set Term.AstConfig.NodeType, or provide default values in AstContext.
Although the parse tree and AST are coming up normally. I have given Ast node type for all nonterminals except transient ones. In each Ast node, I have overridden Init and Doevaluate. DoEvaluate contain only the stub and Init contains the structure of each node type. Is there anything that I am missing, or should I implement the DoEvaluate to get rid of this error?
Coordinator
Sep 12, 2013 at 5:38 PM
can you pls post your final grammar? might be Irony's bu.. feature... :)
Sep 13, 2013 at 3:32 AM
Edited Sep 13, 2013 at 3:36 AM
    public class STGrammar : InterpretedLanguageGrammar
    {
        public STGrammar()
            : base(caseSensitive: false)
        {
            LanguageFlags = LanguageFlags.CreateAst;

            # region Terminals
            Terminal N = new NumberLiteral("number");
            Terminal V = new IdentifierTerminal("variable");
            KeyTerm ASSIGNMENT = ToTerm(":=");
            KeyTerm LEFT_PAREN = ToTerm("(");
            KeyTerm RIGHT_PAREN = ToTerm(")");
            KeyTerm TERMINATE = ToTerm(";");
            KeyTerm LEFT_SQ = ToTerm("[");
            KeyTerm RIGHT_SQ = ToTerm("]");
            KeyTerm DOT = ToTerm(".");
            KeyTerm COMMA = ToTerm(",");
            KeyTerm COLON = ToTerm(":");
            KeyTerm RETURN = ToTerm("return");
            KeyTerm EXIT = ToTerm("exit");
            KeyTerm AND = ToTerm("and");
            KeyTerm OR = ToTerm("or");
            KeyTerm XOR = ToTerm("xor");
            KeyTerm NOT = ToTerm("not");
            KeyTerm FOR = ToTerm("for");
            KeyTerm DO = ToTerm("do");
            KeyTerm END_FOR = ToTerm("end_for");
            KeyTerm BY = ToTerm("by");
            KeyTerm WHILE = ToTerm("while");
            KeyTerm END_WHILE = ToTerm("end_while");
            KeyTerm UNTIL = ToTerm("until");
            KeyTerm REPEAT = ToTerm("repeat");
            KeyTerm END_REPEAT = ToTerm("end_repeat");
            KeyTerm IF = ToTerm("if");
            KeyTerm THEN = ToTerm("then");
            KeyTerm ELSIF = ToTerm("elsif");
            KeyTerm ELSE = ToTerm("else");
            KeyTerm END_IF = ToTerm("end_if");
            KeyTerm TYPE = ToTerm("type");
            KeyTerm END_TYPE = ToTerm("end_type");
            KeyTerm ARRAY = ToTerm("array");
            KeyTerm STRUCT = ToTerm("struct");
            KeyTerm END_STRUCT = ToTerm("end_struct");
            KeyTerm TO = ToTerm("to");
            KeyTerm VAR = ToTerm("var");
            KeyTerm END_VAR = ToTerm("end_var");
            KeyTerm OF = ToTerm("of");
            KeyTerm CASE = ToTerm("case");
            KeyTerm END_CASE = ToTerm("end_case");
            KeyTerm OLD = ToTerm("old");
            KeyTerm NEW = ToTerm("new");

            var comment1 = new CommentTerminal("comment1", "(*", "*);", "*)");
            this.NonGrammarTerminals.Add(comment1);
            var comment2 = new CommentTerminal("comment2", "(#", "#);", "#)");
            this.NonGrammarTerminals.Add(comment2);
            #endregion Terminals

            #region Nonterminals
            NonTerminal actionBlock = new NonTerminal("actionBlock", typeof(ActionBlock));
            NonTerminal action = new NonTerminal("action", typeof(DummyNode));
            NonTerminal assignmentExpr = new NonTerminal("assignment", typeof(AssignmentStNode));
            NonTerminal arrayOrStruct = new NonTerminal("array", typeof(ArrayNode));
            NonTerminal arrayOrStructRHS = new NonTerminal("arrayOrStructRHS", typeof(ArrayRhsNode));
            NonTerminal arrayOrStructRHSs = new NonTerminal(".", typeof(ArrayRHSsNode));
            NonTerminal forStmt = new NonTerminal("for", typeof(ForNode));
            NonTerminal range = new NonTerminal("range", typeof(RangeNode));
            NonTerminal identifier = new NonTerminal("identifier", typeof(DummyNode));
            NonTerminal byExpr = new NonTerminal("by", typeof(DummyNode));
            NonTerminal whileStmt = new NonTerminal("while", typeof(WhileNode));
            NonTerminal repeatStmt = new NonTerminal("repeat", typeof(WhileNode));
            NonTerminal ifStmt = new NonTerminal("if", typeof(IfStNode));
            NonTerminal elsIfStmt = new NonTerminal("elsif", typeof(ElsifNode));
            NonTerminal elsIfBlock = new NonTerminal("elsif", typeof(DummyNode));
            NonTerminal elsePart = new NonTerminal("else", typeof(ElsifNode));
            NonTerminal argument = new NonTerminal("arg", typeof(ArgNode));
            NonTerminal returnValue = new NonTerminal("returnValue", typeof(ArgNode));
            NonTerminal arguments = new NonTerminal("arg", typeof(ArgNode));
            NonTerminal functionArgs = new NonTerminal("functionArgs", typeof(ArgNode));
            NonTerminal funCall = new NonTerminal("function", typeof(FunctionCallNode));
            NonTerminal caseStmt = new NonTerminal("case", typeof(CaseStmtNode));
            NonTerminal caseBlock = new NonTerminal("cases", typeof(CaseBlockNode));
            NonTerminal controlFlow = new NonTerminal("control flow", typeof(DummyNode));
            NonTerminal declarationBlock = new NonTerminal("declare", typeof(BlockNode));
            NonTerminal declarationList = new NonTerminal("declarelist", typeof(VarListNode));
            NonTerminal declarationStmt = new NonTerminal("declareStmt", typeof(VarStmtNode));
            NonTerminal acceptList = new NonTerminal("acceptList", typeof(DummyNode));
            NonTerminal dmList = new NonTerminal("dmList", typeof(DummyNode));
            NonTerminal varBlock = new NonTerminal("variable block", typeof(BlockNode));
            NonTerminal varList = new NonTerminal("varlist", typeof(VarListNode));
            NonTerminal varStmt = new NonTerminal("varStmt", typeof(VarStmtNode));
            NonTerminal term = new NonTerminal("term");
            NonTerminal binExpr = new NonTerminal("binExpr", typeof(BinaryOperationNode));
            NonTerminal parExpr = new NonTerminal("parExpr");
            NonTerminal unExpr = new NonTerminal("unExpr", typeof(UnaryOperationNode));
            NonTerminal unOp = new NonTerminal("unOp");
            NonTerminal binOp = new NonTerminal("binOp", "operator");
            NonTerminal expr = new NonTerminal("expr");
            NonTerminal old_value = new NonTerminal("old_value", typeof(DummyNode));
            NonTerminal new_value = new NonTerminal("new_value", typeof(DummyNode));
            NonTerminal value = new NonTerminal("value");
            NonTerminal elseOpt = new NonTerminal("elseOpt", typeof(DummyNode));
            NonTerminal caseLine = new NonTerminal("caseline", typeof(DummyNode));
            #endregion Nonterminals
Sep 13, 2013 at 3:37 AM
            #region BNFRules
            actionBlock.Rule =
                MakePlusRule(actionBlock, action);
            action.Rule =
                assignmentExpr + TERMINATE
                | arrayOrStruct + TERMINATE
                | forStmt + TERMINATE
                | whileStmt + TERMINATE
                | repeatStmt + TERMINATE
                | ifStmt + TERMINATE
                | funCall + TERMINATE
                | caseStmt + TERMINATE
                | controlFlow + TERMINATE
                | declarationBlock + TERMINATE
                | varBlock + TERMINATE
                ;
            assignmentExpr.Rule =
                identifier + ASSIGNMENT + expr
                ;
            arrayOrStruct.Rule =
                V + arrayOrStructRHSs
                ;
            arrayOrStructRHS.Rule =
                DOT + V
                | LEFT_SQ + V + RIGHT_SQ
                | LEFT_SQ + N + RIGHT_SQ
                ;
            arrayOrStructRHSs.Rule =
                MakePlusRule(arrayOrStructRHSs, arrayOrStructRHS)
                ;
            forStmt.Rule =
                FOR + range + DO + actionBlock + END_FOR
                ;
            range.Rule =
                LEFT_PAREN + identifier + ASSIGNMENT + expr + TO + expr + byExpr + RIGHT_PAREN
                | identifier + ASSIGNMENT + expr + TO + expr + byExpr
                ;
            identifier.Rule =
                V
                | arrayOrStruct
                | old_value
                | new_value
                ;
            byExpr.Rule =
                BY + expr
                | Empty
                ;
            whileStmt.Rule =
                WHILE + expr + DO + actionBlock + END_WHILE
                ;
            repeatStmt.Rule =
                REPEAT + actionBlock + UNTIL + expr + END_REPEAT
                ;
            ifStmt.Rule =
                IF + expr + THEN + actionBlock + elsePart + END_IF
                ;
            elsIfBlock.Rule =
                MakePlusRule(elsIfBlock, elsIfStmt)
                ;
            elsIfStmt.Rule =
                ELSIF + expr + THEN + actionBlock
                ;
            elsePart.Rule =
                PreferShiftHere() + elsIfBlock + ELSE + actionBlock
                | PreferShiftHere() + elsIfBlock
                | PreferShiftHere() + ELSE + actionBlock
                | Empty
                ;
            argument.Rule =
                assignmentExpr
                | expr
                ;
            arguments.Rule =
                argument
                | returnValue
                ;
            returnValue.Rule =
                identifier + ToTerm("=>") + expr
                | identifier + ToTerm("=>") + expr
                ;
            functionArgs.Rule =
                MakeStarRule(functionArgs, COMMA, arguments)
                ;
            funCall.Rule =
                V + LEFT_PAREN + functionArgs + RIGHT_PAREN
                ;
            controlFlow.Rule =
                EXIT
                | RETURN
                ;
            declarationBlock.Rule =
                TYPE + declarationList + END_TYPE
                ;
            declarationList.Rule =
                MakePlusRule(declarationList, declarationStmt)
                ;
            declarationStmt.Rule =
                V + COLON + ARRAY + LEFT_SQ + V + ToTerm("..") + V + RIGHT_SQ + OF + V + TERMINATE
                | V + COLON + STRUCT + dmList + END_STRUCT + TERMINATE
                | V + COLON + LEFT_PAREN + acceptList + RIGHT_PAREN + TERMINATE
                ;
            dmList.Rule =
                MakePlusRule(dmList, V + COLON + V + TERMINATE)
                ;
            acceptList.Rule =
                MakePlusRule(acceptList, COMMA, V)
                ;
            varBlock.Rule =
                VAR + varList + END_VAR
                ;
            varList.Rule =
                MakePlusRule(varList, varStmt)
                ;
            varStmt.Rule =
                V + COLON + ARRAY + LEFT_SQ + V + ToTerm("..") + V + RIGHT_SQ + OF + V + TERMINATE
                | V + COLON + V + TERMINATE
                | V + COLON + V + ASSIGNMENT + value + TERMINATE
                ;
            expr.Rule = term | unExpr | binExpr;
            term.Rule = N | parExpr | funCall | identifier;
            parExpr.Rule = LEFT_PAREN + expr + RIGHT_PAREN;
            unExpr.Rule = unOp + term + ReduceHere();
            unOp.Rule = ToTerm("-") | NOT;
            binExpr.Rule = expr + binOp + expr;
            binOp.Rule = ToTerm("+") | "-" | "*" | "/" | "^" | "=" | "<" | "<=" | ">" | ">=" | "<>" | AND | OR | XOR | "&gt;" | "&lt;" | "&gt;=" | "&lt;=" | "&lt;&gt;";
            caseStmt.Rule =
                CASE + expr + OF + caseBlock + elseOpt + END_CASE
                ;
            caseBlock.Rule =
                MakePlusRule(caseBlock, caseLine)
                ;
            caseLine.Rule =
                value + COLON + action
                ;
            old_value.Rule =
                V + PreferShiftHere() + COLON + OLD
                ;
            new_value.Rule =
                V + PreferShiftHere() + COLON + NEW
                ;
            value.Rule =
                identifier
                | N
                ;
            elseOpt.Rule =
                ELSE + actionBlock
                | Empty
                ;

            this.Root = actionBlock;
            #endregion BNFRules
Sep 13, 2013 at 3:38 AM
            #region OperatorPrecedence
            RegisterOperators(10, ":=");
            RegisterOperators(15, "=", "<", "<=", ">", ">=", "<>", "and", "or", "xor", "&gt;", "&lt;", "&gt;=", "&lt;=", "&lt;&gt;");
            RegisterOperators(20, "+", "-");
            RegisterOperators(30, "*", "/", "mod");
            RegisterOperators(60, Associativity.Right, "^");
            RegisterOperators(70, "(");
            #endregion OperatorPrecedence

            MarkPunctuation(",", ";", "[", "]", ")", "(", "..", ".", ":", "by", "new", "old");
            RegisterBracePair("(", ")");
            this.MarkTransient(action, arrayOrStructRHS, identifier, byExpr, term, parExpr, unOp, binOp, expr, value, arguments);
        }
This is my grammar.
Coordinator
Sep 17, 2013 at 7:47 AM
There is a bug in Irony in the way it reports the error, it should say the names of nonterminals that have missing AstNode type.
And after fixing this (I will push the fix shortly), here's your problem. It is in the statement:

dmList.Rule =
      MakePlusRule(dmList, V + COLON + V + TERMINATE)
The second argument of MakePlusRule is automatically converted to NonTerminal - that's the way it works - and this NT has no AstNode type. You should declare this NonTerminal explicitly and assign AstNode type to it
Coordinator
Sep 17, 2013 at 7:51 AM
Here's the fix for the parser, just in case you need it asap. File AstBuilder.cs, method VerifyLanguageData, at the very end, replace the AddMessage call with the following:
    Context.AddMessage(ErrorLevel.Error, SourceLocation.Empty, Resources.ErrNodeTypeNotSetOn, string.Join(", " , missingList));
Just checked with explicitly declared nonterminal for this dmList-element expr, it no longer gives an error.
Marked as answer by sreejas on 4/22/2015 at 1:34 AM
Sep 17, 2013 at 7:57 AM
Thank you for the reply. It is working fine now.