Help with indentation-based grammar

Jun 11, 2014 at 11:08 PM
I'm trying to create a rather simple DSL that's indentation-based.
I would like to be able to parse the following code:
@default "test"
@test default ()

// this is also a comment

test:
    Log.debug 5
    Log.info "test" // end of line comment

test2:
    @depends [test]
    Log.info "this is friggin test2"
However, I'm having some problems. When I run the code above against my parser I get a syntax error saying it expected (, [, identifier, string, number, basically a Value terminal, at the begining of the line with test2:.
Any help with getting this to work would be much appreciated.

The grammar-file I have looks like this:
using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Irony.Parsing;

namespace Kake.Parser
{
    [Language("Kakefile", "0.1", "Kake DSL")]
    class KakeGrammar : Grammar
    {
        public KakeGrammar()
            : base(caseSensitive: true)
        {
            // 1. Terminals
            var number = TerminalFactory.CreatePythonNumber("number");
            var identifier = CreateKakeIdentifier("identifier");
            var comment = new CommentTerminal("comment", "//", "\n", "\r");

            //comment must to be added to NonGrammarTerminals list; it is not used directly in grammar rules,
            // so we add it to this list to let Scanner know that it is also a valid terminal. 
            NonGrammarTerminals.Add(comment);
            var comma = ToTerm(",");
            var colon = ToTerm(":");
            var semi = ToTerm(";");
            var dot = ToTerm(".");
            var ws = ToTerm(" ");
            var str = TerminalFactory.CreateCSharpString("string");
            var num = TerminalFactory.CreateCSharpNumber("number");

            // 2. Non-terminals
            var Expr = new NonTerminal("Expr");
            var Exprs = new NonTerminal("Exprs");
            var ParExpr = new NonTerminal("ParExpr");

            var AssignmentExpr = new NonTerminal("AssignmentExpr");
            var Targets = new NonTerminal("Targets");
            var TargetList = new NonTerminal("TargetList");
            var TargetTuple = new NonTerminal("TargetTuple");

            var Call = new NonTerminal("Call");
            var CallTarget = new NonTerminal("CallTarget");
            var CallArg = new NonTerminal("CallArg");
            var CallArgs = new NonTerminal("CallArgs");

            var Value = new NonTerminal("Value");
            var Array = new NonTerminal("Array");
            var ArrayElements = new NonTerminal("ArrayElements");

            var Meta = new NonTerminal("Meta");
            var MetaList = new NonTerminal("MetaList");

            var Target = new NonTerminal("Target");
            var TargetArgs = new NonTerminal("TargetArgs");
            var TargetBody = new NonTerminal("TargetBody");
            var DocTargets = new NonTerminal("DocTargets");

            var Document = new NonTerminal("Document");

            // 3. BNF rules
            Expr.Rule = ParExpr | identifier | Value | AssignmentExpr | Call;
            ParExpr.Rule = "(" + (Expr | Empty) + ")";
            Exprs.Rule = MakePlusRule(Exprs, Eos, Expr);

            AssignmentExpr.Rule = Targets + "=" + Expr;

            TargetList.Rule = MakePlusRule(TargetList, comma, Targets);
            TargetTuple.Rule = "(" + TargetList + ")";
            Targets.Rule = identifier | TargetTuple;

            Call.Rule = CallTarget + CallArgs;
            CallTarget.Rule = MakePlusRule(CallTarget, dot, identifier);
            CallArg.Rule = ParExpr | identifier | Value;
            CallArgs.Rule = MakePlusRule(CallArgs, CallArg);

            Value.Rule = Array | str | num;
            Array.Rule = ToTerm("[") + ArrayElements + "]";
            ArrayElements.Rule = MakeStarRule(ArrayElements, semi, Value);

            Meta.Rule = ToTerm("@") + identifier + CallArgs + Eos;
            MetaList.Rule = MakeStarRule(MetaList, Meta);

            Target.Rule = identifier + TargetArgs + colon + Eos + Indent + TargetBody + Dedent + Eos;
            TargetArgs.Rule = MakeStarRule(TargetArgs, identifier);
            TargetBody.Rule = MetaList + Exprs;
            DocTargets.Rule = MakeStarRule(DocTargets, Target);

            Document.Rule = MetaList + DocTargets + Eof;

            Root = Document;

            // 4. Misc
            MarkPunctuation("(", ")", "[", "]", ":");
            RegisterBracePair("(", ")");
            RegisterBracePair("[", "]");
        }

        static IdentifierTerminal CreateKakeIdentifier(string name)
        {
            var id = new IdentifierTerminal(name);
            id.CharCategories.Add(UnicodeCategory.DashPunctuation);
            return id;
        }

        public override void CreateTokenFilters(LanguageData language, TokenFilterList filters)
        {
            var outlineFilter = new CodeOutlineFilter(language.GrammarData, OutlineOptions.ProduceIndents | OutlineOptions.CheckBraces, null);
            filters.Add(outlineFilter);
        }
    }
}
Coordinator
Jun 12, 2014 at 3:33 AM
I think the problem is empty line - your grammar does not allow this, at least the Targets is a list of Target, which may not be empty. Modify grammar (change Target?) to allow empty lines
Jun 12, 2014 at 9:24 AM
Removing the empty line does not help. Also, there is nowhere in my grammar that allows for empty lines, yet the empty lines everywhere else works. I expect the tokenization removes these. Is there some way I can get a token-stream out of my Grammar, that way I could inspect the tokens and see if it creates Indent, Dedent and Eos correctly. Cause it tells me it's expecting a identifier at (10,1) which is an identifier, so I'm guessing there has to be a virtual token there.
Jun 12, 2014 at 9:31 AM
Ah, I figured it out :)
Changing Exprs.Rule to MakeListRule(Exprs, Eos, Expr, TermListOptions.PlusList | TermListOptions.AllowTrailingDelimiter); resolved the problem (and made new ones, but those are easier to fix).