Identation Question

May 2, 2012 at 8:58 PM
Edited May 2, 2012 at 8:59 PM

Hello,

I have a language that use indentation for code validation. Here is a sample :

 

? Identation lvl 0
	- Indentation lvl1
	- Indentation lvl1
		? Indentation lvl 2
		? Indentation lvl 2
		? Indentation lvl 2
			- Indentation lvl 3
			- Indentation lvl 3

 

Level of indentation does not have a fixed limit. I'm trying to use Indent as MiniPython do, but i don't understand how it works, and where to define indentation in my sample is based on tabulation or four spaces. Moreover, the language i want to parse must not eat whitespaces, so i override SkipWhiteSpaces.

When i try something like that :

var HEADERPATTERN = new NonTerminal("HEADERPATTERN");
HEADERPATTERN.Rule = "? " + HEADER + NewLine + Indent + Block + Dedent;

Parsing fails on line 2, waiting for "Indent" , so i don't understand how it works.

Can someone explain me please? 

EDIT : this is my second post on this language, first is here talking about tabulations in terminals http://irony.codeplex.com/discussions/352482

Coordinator
May 2, 2012 at 9:06 PM

are you adding CodeOutlineToken filter to token filters (like MiniPython does)?

May 2, 2012 at 9:12 PM

Yes, Parsing error fails on end of first line waiting for Indent

Coordinator
May 2, 2012 at 9:15 PM

hard to say, try to debug into CodeOutlineFilter, see if it produces Indent token, and if not - try to see why

May 2, 2012 at 9:19 PM
Edited May 2, 2012 at 9:31 PM

Just in case, this is a realy basic implementation of the language

[Language("LanguageIndentTest")]
    public class LanguageIndentTest : Grammar
    {
        public LanguageIndentTest()
            : base(true)
        {
            var lineComment = new CommentTerminal("line_comment", "//", "\n", "\r\n");
            this.NonGrammarTerminals.Add(lineComment);

            IdentifierTerminal IDENTIFIER = new IdentifierTerminal("IDENTIFIER");

            var HEADER = new NonTerminal("HEADER");
            HEADER.Rule = MakePlusRule(HEADER, ToTerm(" "), IDENTIFIER);

            var Block = new NonTerminal("Block");
            Block.Rule = "- " + IDENTIFIER;

            var HEADERPATTERN = new NonTerminal("HEADERPATTERN");
            HEADERPATTERN.Rule = "? " + HEADER + NewLine + Indent + Block + Dedent + NewLine + Eof;

            this.Root = HEADERPATTERN;
        }

        public override void CreateTokenFilters(LanguageData language, TokenFilterList filters)
        {
            var outlineFilter = new CodeOutlineFilter(language.GrammarData,
              OutlineOptions.ProduceIndents | OutlineOptions.CheckBraces, ToTerm(@"\")); // "\" is continuation symbol
            filters.Add(outlineFilter);
        }

        public override void SkipWhitespace(ISourceStream source)
        {
            return;
        }
Coordinator
May 4, 2012 at 6:12 PM

1. First, remove override of SkipWhitespace - you do need default space processing.

2. Do not use space as a terminal (like in HEADER rule); HEADER is identifier list, so simply define it as 

  HEADER.Rule = MakePlusRule(HEADER, IDENTIFIER);

3. Add @"\" to NonGrammarTerminals, as a message on Grammar Errors page tells you

4. Do not use NewLine anywhere, use Eos terminal instead (look at MiniPython grammar)

5. Finally, you grammar rules do not match your input sample at all. Look closely, you do not allow more than one line in a block; line starting with "?" is possible only at the beginning of the file, and so on. Once you've fixed 1-4, start fixing your grammar - look at parser trace and token list in Grammar Explorer, try to figure out what's missing

Roman

Jun 11, 2012 at 2:49 PM

Hello,

Thank you a lot for your advices, i've got now something that works. But i've got another question. I want to include specific error messages, because with this code

 

[Language("AthenaIndentTest")]
    public class AthenaIndentTest : Grammar
    {
        public AthenaIndentTest()
            : base(true)
        {

            var lineComment = new CommentTerminal("line_comment", "//", "\n", "\r\n");
            this.NonGrammarTerminals.Add(lineComment);
            this.NonGrammarTerminals.Add(ToTerm(@"\"));

            IdentifierTerminal IDENTIFIER = new IdentifierTerminal("IDENTIFIER");
            NonTerminal Stmt = new NonTerminal("Statement");
            NonTerminal StmtList = new NonTerminal("StmtList");
            NonTerminal TEMPLATE_LIST = new NonTerminal("TEMPLATE_LIST");
            NonTerminal TEMPLATE = new NonTerminal("TEMPLATE");
            NonTerminal PATTERN_LIST = new NonTerminal("PATTERN_LIST");
            NonTerminal PATTERN = new NonTerminal("PATTERN");
            NonTerminal SCENARIO = new NonTerminal("SCENARIO");
            NonTerminal SUB_SCENARIOS = new NonTerminal("SUB_SCENARIOS");
            NonTerminal SUB_SCENARIO = new NonTerminal("SUB_SCENARIO");

            StmtList.Rule = MakePlusRule(StmtList, Stmt);
            Stmt.Rule = SCENARIO;
            SCENARIO.Rule = PATTERN_LIST + Indent + TEMPLATE_LIST + ((Indent + SUB_SCENARIOS + Dedent) | Empty) + Dedent;
            PATTERN_LIST.Rule = MakePlusRule(PATTERN_LIST, PATTERN);
            PATTERN.Rule = ToTerm("?") + IDENTIFIER + Eos;
            TEMPLATE_LIST.Rule = MakePlusRule(TEMPLATE_LIST, TEMPLATE);
            TEMPLATE.Rule = ToTerm("-") + IDENTIFIER + Eos;
            SUB_SCENARIOS.Rule = MakeStarRule(SUB_SCENARIOS, SUB_SCENARIO);
            SUB_SCENARIO.Rule = PATTERN_LIST + Indent + TEMPLATE_LIST + Dedent;

            this.Root = StmtList;
        }

        public override void CreateTokenFilters(LanguageData language, TokenFilterList filters)
        {
            var outlineFilter = new CodeOutlineFilter(language.GrammarData,
              OutlineOptions.ProduceIndents | OutlineOptions.CheckBraces, ToTerm(@"\")); // "\" is continuation symbol
            filters.Add(outlineFilter);
        }
    }

 

 

And this text to parse :

 

? AIFAIM
    - tutu
    - titi
        ? PATTERN
        ? PATTERN
            - TEMPLATE
            - TEMPLATE
        ? PATTERN
    
    
? AIFAIM
    - tutu
    - titi

 

I'm getting INDENT expected, and i would like to put an error like this : Missing template

Any idea to do this ?

Thanks

Coordinator
Jun 11, 2012 at 5:14 PM

you can override Grammar.ConstructParserErrorMessage method, and provide your own "interpretation" of the error.