Irony - .NET Language Implementation Kit.

New Post: Looking for a tutorial

sherif234 — Thu, 23 May 2013 08:38:30 GMT

Hello,

Thank you for your reply. I am trying to generate an sql script based on the user input. The old tutorials were really helpful, but the new source code seems a bit complicated without any documentation. Basically, I had the following:

A compiler class with the following function:

   public static string Compile(string sourceCode)
    {
        // create a compiler from the grammar
        FLGrammar grammar = new FLGrammar();
        LanguageCompiler compiler = new LanguageCompiler(grammar);

        // Attempt to compile into an Abstract Syntax Tree. Because FLGrammar
        // defines the root node as ProgramNode, that is what will be returned.
        // This happens to implement ILangGenerator, which is what we need.
        ILangGenerator program = (ILangGenerator)compiler.Parse(sourceCode);
        if (program == null || compiler.Context.Errors.Count > 0)
        {
            // Didn't compile.  Generate an error message.
            SyntaxError error = compiler.Context.Errors[0];
            string location = string.Empty;
            if (error.Location.Line > 0 && error.Location.Column > 0)
            {
                location = "Line " + (error.Location.Line + 1) + ", column " + (error.Location.Column + 1);
            }
            string message = location + ": " + error.Message + ":" + Environment.NewLine;
            message += sourceCode.Split('\n')[error.Location.Line];

            throw new CompilationException(message);
        }

        // now just instruct the compilation of to javascript
        StringBuilder js = new StringBuilder();
        program.GenerateScript(js);
        return js.ToString();

    }

A grammar class:

    public FLGrammar()
    {

        #region Initial setup of the grammar

        this.CaseSensitive = false;

        // define all the non-terminals
        var program = new NonTerminal("program", typeof(ProgramNode));
        var statementList = new NonTerminal("statementList", typeof(StatementListNode));
        var condition = new NonTerminal("statement", typeof(ConditionNode));

        var oper = new NonTerminal("operator", typeof(OperatorNode));

        // define all the terminals
        var variable = new IdentifierTerminal("variable");
        variable.AddKeywords("where","set", "to", "if", "freight", "cost", "is", "loop", "through", "order");
        var number = new NumberLiteral("number");
        var stringLiteral = new StringLiteral("string", "\"", ScanFlags.None);

        // remove uninteresting nodes from the AST (note: in current version of Irony,
        // keywords added to the variable cannot be registered as punctuation).
        this.RegisterPunctuation(";", "[", "]", "(", ")");

        // specify the non-terminal which is the root of the AST
        this.Root = program;

        #endregion

        #region Define the grammar

        //<Program> ::= <StatementList> <FreightDeclaration>
        program.Rule = statementList;

        //<StatementList> ::= <Statement>*
        statementList.Rule = Symbol("where") + condition;

        //<condition>::= <string> <operator> <string> | <condition> <operator> <condition> | “(“  <condition> “)”

        //<Statement> ::= <SetVariable> ";" | <IfStatement> | <OrderLoop> | <Expression> ";"
        condition.Rule = condition + oper + condition | "(" + condition + ")" | variable | 
            number | stringLiteral;




        //<BinaryOperator> ::= "+" | "-" | "*" | "/" | "<" | ">" | "<=" | ">=" | "is"
        oper.Rule = Symbol("+") | "-" | "*" | "/" | "<" | ">" | "<=" | ">=" | "is" | 
            "=" | "like" | "and" | "or";


        #endregion

    }

A class for each node, to generate the related SQL statements. Each class inherits AstNode, and implements a Generator interface which has a virtual function for generating scripts.

With the new Irony source code, I have looked at the ExpressionEvaluatorGrammar example. I can see three classes for the grammar:
ExpressionEvaluator, ExpressionEvaluatorGrammar & ExpressionEvaluatorRuntime. In addition to that, there are are classes for AstNode. This time, these classes only inherit from AstNode. I dont know where should I implement the logic for generating the SQL script. Should it be in the overriden Init function?

Also it seems the ExpressionEvaluator example makes use of several LanguageRuntime classes: LanguageRuntime, LanguageRuntime_Binding, LanguageRuntime_OpDispatch, LanguageRuntime_OpDispatch_Init. Do I need to implement all these three classes to achieve what I am looking for?

Please help. Thank you.

New Post: Looking for a tutorial

MindCore — Wed, 22 May 2013 19:13:14 GMT

Currently, the best tutorials are the sample projects in the source code. Unfortunately, there are no step-by-step tutorials as Roman has been swamped, but I'm sure he would accept user submissions with regards to this.

Sounds like you are already off to a good start with code previously designed around the old Irony architecture. I am not sure how to your request for help on the current Irony without some context to your issues. If you could, please post some of your old code, and the community may be able to help.

New Post: Looking for a tutorial

sherif234 — Wed, 22 May 2013 10:56:03 GMT

Hello,

I have been struggling to find a step by step tutorial for the latest version of Irony. I can only find the two posts created by Daniel Flower on Code project. Yes they are good, but unfortunately, it is only applicable to the old version of Irony.

I am trying to generate/build a string (sql statements) based on the user input. Please help. Thank you.

New Post: System.NullReferenceException in BuildAst

jarin — Sat, 18 May 2013 17:52:28 GMT

Hi Roman, thanks for your reply.

I've started by stripping my grammar to bare minimum (only numbers and operators) but had the same issue. During debugging I found that DefaultLiteralNodeType was null and found this discussion that explained what was going on. Overriding BuildAst() did the trick and I got the basic grammar working. I'll start adding more stuff to my grammar and see what happens. At least I've got slightly better understanding of how it all works now.

Thanks!

New Post: System.NullReferenceException in BuildAst

rivantsov — Sat, 18 May 2013 04:11:13 GMT

at first look, you are missing AST node type for colname terminal. Stop in debugger on exception and look around on data involved (which non-terminal/terminal is there). But this is my guess - colname. You have to provide AstNode implementation - a class that knows how to interpret the colname at runtime.
Let me know if you're stuck, I may be able to investigate in details later
Roman

New Post: System.NullReferenceException in BuildAst

jarin — Fri, 17 May 2013 13:24:11 GMT

I've got very simple grammar adapted from "Writing a calculator in C# using Irony" sample (see below) and parsing works fine, however when I add

LanguageFlags = LanguageFlags.CreateAst;

to generate AST I get following error in Parser Output tab:

AstNodeType or AstNodeCreator is not set on non-terminals: Irony.Parsing.BnfTermList. Either set Term.AstConfig.NodeType, or provide default values in AstContext.

and an exception in Irony Grammar Explorer:

System.NullReferenceException: Object reference not set to an instance of an object.
   at Irony.Ast.AstBuilder.BuildAst(ParseTreeNode parseNode) in C:\Programming\Irony_2013_03_10\Irony_2013_03_10\Irony\Ast\AstBuilder.cs:line 97
   at Irony.Ast.AstBuilder.BuildAst(ParseTreeNode parseNode) in C:\Programming\Irony_2013_03_10\Irony_2013_03_10\Irony\Ast\AstBuilder.cs:line 86
   at Irony.Ast.AstBuilder.BuildAst(ParseTree parseTree) in C:\Programming\Irony_2013_03_10\Irony_2013_03_10\Irony\Ast\AstBuilder.cs:line 38
   at Irony.Parsing.Grammar.BuildAst(LanguageData language, ParseTree parseTree) in C:\Programming\Irony_2013_03_10\Irony_2013_03_10\Irony\Parsing\Grammar\Grammar.cs:line 499
   at Irony.Parsing.Parser.Parse(String sourceText, String fileName) in C:\Programming\Irony_2013_03_10\Irony_2013_03_10\Irony\Parsing\Parser\Parser.cs:line 88
   at Irony.GrammarExplorer.fmGrammarExplorer.ParseSample() in C:\Programming\Irony_2013_03_10\Irony_2013_03_10\Irony.GrammarExplorer\fmGrammarExplorer.cs:line 359
   at Irony.GrammarExplorer.fmGrammarExplorer.btnParse_Click(Object sender, EventArgs e) in C:\Programming\Irony_2013_03_10\Irony_2013_03_10\Irony.GrammarExplorer\fmGrammarExplorer.cs:line 507
   at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
   at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
   at System.Windows.Forms.Control.WndProc(Message& m)
   at System.Windows.Forms.ButtonBase.WndProc(Message& m)
   at System.Windows.Forms.Button.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

My grammar is:

public class TestGrammar1 : Grammar
    {
        public TestGrammar1 () : base(false)
        {
            var number = new NumberLiteral("number", NumberOptions.AllowSign | NumberOptions.AllowStartEndDot);
            var fnname = new IdentifierTerminal("fnname");
            
            var colname = new StringLiteral("colname");
            colname.AddStartEnd("[", "]", StringOptions.None);

            var expression = new NonTerminal("expression");
            var binexpr = new NonTerminal("binexpr", typeof(BinaryOperationNode));
            var parexpr = new NonTerminal("parexpr");
            var fncall = new NonTerminal("fncall", typeof(FunctionCallNode));
            var binop = new NonTerminal("binop", "operator");

            expression.Rule = parexpr | binexpr | number | colname | fncall;
            parexpr.Rule = "(" + expression + ")";
            binexpr.Rule = expression + binop + expression;
            binop.Rule = ToTerm("+") | "-" | "/" | "*";
            fncall.Rule = fnname + "(" + expression + ")";
            this.Root = expression;

            MarkPunctuation("(",")");

            RegisterOperators(1, "+", "-");
            RegisterOperators(2, "*", "/");
            AddOperatorReportGroup("operator");
            
            MarkTransient(parexpr, expression, binop);
            LanguageFlags = LanguageFlags.CreateAst;
        }
    }

My test expression:

sum([a])

Can somebody please help me figure out what's wrong with my grammar?
Thanks!

New Post: Fatal error in code colorizer.Colorizing has been disabled

rivantsov — Tue, 14 May 2013 21:24:21 GMT

hi
sorry, do not have much time right now, it would be a few days
Roman

New Post: Fatal error in code colorizer.Colorizing has been disabled

nadvi — Sun, 12 May 2013 15:10:37 GMT

As-Salam-wa-alicum

Sir,

Thank you very much for your effort but the way you advice need so many work at comiletime to determine the actual thing because of its generality
and it is possible to make it a little more specific.I did it but when I try to refactor emofAttributeDiclaration,functionSignatureBody and ArrayType so
many conflict come.Please help me here.My grammar file and the test file is here By the way do you have the original grammar
file on which you do work to fix conflict?

Thank you very much again for reading my mail by spending your valuable time and working on it.

With regards
Nadvi

New Post: "Reduce-reduce" errors eat my lunch; we really need some context on them

brantheman — Fri, 26 Apr 2013 21:45:58 GMT

Thanks for the clue! I've got it all figured out. Here's a little documentation for anyone else dealing with Reduce-Reduce errors:

You can only have one of each RValue (right-hand side of the arror value) in the "Reduce Items" section. If you have more than one it can't decide which non-terminal to use in a given scenario. (It likely doesn't matter which one gets used; I guess Irony doesn't want to take any chances.)

This won't work as all child trees support an Empty:

foreach(var group in groups)
{
   ....
   var myRule = ....;
   myRule.Rule = MakeStarRule(myRule, eachRowOfMyRule);

   if(program.Rule == null)
       program.Rule = myRule;
   else program.Rule |= myRule;
}

You have to do it like this instead:

foreach(var group in groups)
{
   ....
   var myRule = ....;
   if(program.Rule == null)
      myRule.Rule = MakeStarRule(myRule, eachRowOfMyRule);
   else
      myRule.Rule = MakePlusRule(myRule, eachRowOfMyRule);

   if(program.Rule == null)
       program.Rule = myRule;
   else program.Rule |= myRule;
}

New Post: "Reduce-reduce" errors eat my lunch; we really need some context on them

rivantsov — Fri, 26 Apr 2013 20:55:00 GMT

my guess is that you have several optional sections, optionally (!) included, and parser does not know how to interpret an empty file. To say more, I need to see the grammar

New Post: "Reduce-reduce" errors eat my lunch; we really need some context on them

brantheman — Fri, 26 Apr 2013 20:33:40 GMT

Here is the output from Grammar Explorer. I assume that I look in the "Reduce Items" section, but I don't know what to look for there.

State S0 (Inadequate)
  Reduce-reduce conflicts on inputs: EOF LF
  Shift items:
    Program' -> ·Program EOF 
    Program -> ·StartSection.grp1 
    StartSection.grp1 -> ·StartSectionRow.grp1+ 
    StartSectionRow.grp1+ -> ·StartSectionRow.grp1+ StartSectionRow.grp1 
    StartSectionRow.grp1+ -> ·StartSectionRow.grp1 
    StartSectionRow.grp1 -> ·OptionalTelemetryTriples.grp1 LF 
    OptionalTelemetryTriples.grp1 -> ·TelemetryTriples.grp1 
    TelemetryTriples.grp1 -> ·Telemetry 
    Telemetry -> ·until Unnamed0 Telemetry.Operator Telemetry.Value OptionalMetric 
    TelemetryTriples.grp1 -> ·Telemetry 
    Telemetry -> ·until Unnamed1 Telemetry.Operator Telemetry.Value OptionalMetric 
    StartSectionRow.grp1 -> ·ActionTriplet.grp1 OptionalTelemetryTriples.grp1 LF 
    ActionTriplet.grp1 -> ·ActionSetTriples.grp1 
    ActionSetTriples.grp1 -> ·Action 
    Action -> ·set Unnamed2 at Action.Value OptionalMetric 
    ActionSetTriples.grp1 -> ·Action 
    Action -> ·set Unnamed3 at Action.Value OptionalMetric 
    ActionTriplet.grp1 -> ·ActionIncTriples.grp1 
    ActionIncTriples.grp1 -> ·Action 
    Action -> ·inc Unnamed4 by Action.Value OptionalMetric 
    ActionIncTriples.grp1 -> ·Action 
    Action -> ·inc Unnamed5 by Action.Value OptionalMetric 
    ActionTriplet.grp1 -> ·ActionDecTriples.grp1 
    ActionDecTriples.grp1 -> ·Action 
    Action -> ·dec Unnamed6 by Action.Value OptionalMetric 
    ActionDecTriples.grp1 -> ·Action 
    Action -> ·dec Unnamed7 by Action.Value OptionalMetric 
    ActionTriplet.grp1 -> ·ActionMulTriples.grp1 
    ActionMulTriples.grp1 -> ·Action 
    Action -> ·mul Unnamed8 by Action.Value OptionalMetric 
    ActionMulTriples.grp1 -> ·Action 
    Action -> ·mul Unnamed9 by Action.Value OptionalMetric 
    Program -> ·StartSection.grp2 
    StartSection.grp2 -> ·StartSectionRow.grp2+ 
    StartSectionRow.grp2+ -> ·StartSectionRow.grp2+ StartSectionRow.grp2 
    StartSectionRow.grp2+ -> ·StartSectionRow.grp2 
    StartSectionRow.grp2 -> ·OptionalTelemetryTriples.grp2 LF 
    OptionalTelemetryTriples.grp2 -> ·TelemetryTriples.grp2 
    TelemetryTriples.grp2 -> ·Telemetry 
    Telemetry -> ·until Unnamed10 Telemetry.Operator Tail OptionalMetric 
    TelemetryTriples.grp2 -> ·Telemetry 
    Telemetry -> ·until Unnamed11 Telemetry.Operator Telemetry.Value OptionalMetric 
    StartSectionRow.grp2 -> ·ActionTriplet.grp2 OptionalTelemetryTriples.grp2 LF 
    ActionTriplet.grp2 -> ·ActionSetTriples.grp2 
    ActionSetTriples.grp2 -> ·Action 
    Action -> ·set Unnamed12 at Action.Value OptionalMetric 
    ActionTriplet.grp2 -> ·ActionIncTriples.grp2 
    ActionIncTriples.grp2 -> ·Action 
    Action -> ·inc Unnamed13 by Action.Value OptionalMetric 
    ActionTriplet.grp2 -> ·ActionDecTriples.grp2 
    ActionDecTriples.grp2 -> ·Action 
    Action -> ·dec Unnamed14 by Action.Value OptionalMetric 
    ActionTriplet.grp2 -> ·ActionMulTriples.grp2 
    ActionMulTriples.grp2 -> ·Action 
    Action -> ·mul Unnamed15 by Action.Value OptionalMetric 
  Reduce items:
    StartSection.grp1 -> · [EOF]
    OptionalTelemetryTriples.grp1 -> · [LF]
    StartSection.grp2 -> · [EOF]
    OptionalTelemetryTriples.grp2 -> · [LF]
  Transitions: Program->S1, StartSection.grp1->S2, StartSectionRow.grp1+->S3, StartSectionRow.grp1->S4, OptionalTelemetryTriples.grp1->S5, TelemetryTriples.grp1->S6, Telemetry->S7, until->S8, Telemetry->S9, ActionTriplet.grp1->S10, ActionSetTriples.grp1->S11, Action->S12, set->S13, Action->S14, ActionIncTriples.grp1->S15, Action->S16, inc->S17, Action->S18, ActionDecTriples.grp1->S19, Action->S20, dec->S21, Action->S22, ActionMulTriples.grp1->S23, Action->S24, mul->S25, Action->S26, StartSection.grp2->S27, StartSectionRow.grp2+->S28, StartSectionRow.grp2->S29, OptionalTelemetryTriples.grp2->S30, TelemetryTriples.grp2->S31, Telemetry->S32, Telemetry->S33, ActionTriplet.grp2->S34, ActionSetTriples.grp2->S35, Action->S36, ActionIncTriples.grp2->S37, Action->S38, ActionDecTriples.grp2->S39, Action->S40, ActionMulTriples.grp2->S41, Action->S42

Commented Unassigned: crash with rv null in ComputeProductionFlags [9881]

brantheman — Fri, 26 Apr 2013 19:58:15 GMT

I got the crash (shown in the attached screenshot).
Comments: I think you'll have to fix both Op_Plus and Op_Pipe to handle null inputs.

Commented Unassigned: crash with rv null in ComputeProductionFlags [9881]

rivantsov — Fri, 26 Apr 2013 17:27:43 GMT

I got the crash (shown in the attached screenshot).
Comments: yeah, I see the problem. I think the best place to add this is in overload of "+" operator. Will add in the next code update thanks Roman

New Post: Fatal error in code colorizer.Colorizing has been disabled

rivantsov — Fri, 26 Apr 2013 17:16:58 GMT

Man, it's time to take over the thing, rollup your sleeves and start pushing it. I've fixed all the conflicts (the most serious ones), so now you can move in small increments. Just start modifying this parameter definition when it is a 'lambda' to bring back the old code I removed - as far as I understand this is the problem now
Try it, read something about LALR parsing and about conflicts, try it again. Let me know if you're stuck
Roman

New Post: "Reduce-reduce" errors eat my lunch; we really need some context on them

rivantsov — Fri, 26 Apr 2013 17:13:11 GMT

well sorry for your lunch, but - don't quite get what's the problem with the conflict? did you try to double-click on conflict message in Grammar Explorer? It will bring you to the state printout, which show you all.
About NewLineBeforeEOF - this has no impact on parser construction. It is at runtime, when parsing, the scanner injects NewLine before sending EOF to parser, just to adjust for source files that have no final line-break, but the language is line-based.

New Post: "Reduce-reduce" errors eat my lunch; we really need some context on them

brantheman — Thu, 25 Apr 2013 21:31:33 GMT

"Reduce-reduce conflict. State S0, lookaheads: EOF LF. Selected reduce on first production in conflict set. (S0)"

My grammar file is only 170 lines long, but I still have absolutely no idea where to even start with an error like this. Can we put the nearest non-terminal name in the error somewhere?

Also, I thought Irony treated EOF as an alternate for LF if I used this: LanguageFlags |= LanguageFlags.NewLineBeforeEOF. Is that not true? Why would it have a conflict between the two?

New Post: "Reduce-reduce" errors eat my lunch; we really need some context on them

brantheman — Thu, 25 Apr 2013 21:31:33 GMT

Commented Unassigned: crash with rv null in ComputeProductionFlags [9881]

brantheman — Thu, 25 Apr 2013 21:15:54 GMT

I got the crash (shown in the attached screenshot).
Comments: I find that there was a problem in my code. I was using the |= operator on a rule that had not been initialized. It would be nice if that case was handled appropriately. The | operator should return the non-null item out of two.

Commented Unassigned: crash with rv null in ComputeProductionFlags [9881]

brantheman — Thu, 25 Apr 2013 21:10:21 GMT

I got the crash (shown in the attached screenshot).
Comments: At the point of the crash the parent CreateProduction lvalue = {StartSectionRow.grp1}

Commented Unassigned: crash with rv null in ComputeProductionFlags [9881]

brantheman — Thu, 25 Apr 2013 21:08:20 GMT

I got the crash (shown in the attached screenshot).
Comments: I've attached my grammar also.