indent problem

Dec 22, 2008 at 10:22 AM

I defined my syntax, that worked very good.
But now i must ask again about indent.

I made a smale sample:

 

 

public testsprechGrammer(){
NonTerminal compilation_unit = new NonTerminal("compilation_unit");
SymbolTerminal _cmd = Symbol("KOMMANDO");
SymbolTerminal _ind = Symbol("INDENT");
NonTerminal zeile = new NonTerminal("zeile");
NonTerminal indentzeile = new NonTerminal("indentzeile");
zeile.Rule = _cmd +
Grammar.NewLine;
indentzeile.Rule =
Grammar.Indent + _ind;
compilation_unit.Rule = zeile + indentzeile;
TokenFilters.Add(
new CodeOutlineFilter(false));
this.Root = compilation_unit;
}

Than I used this test:
KOMMANDO
 INDENT

I get than this result:

Syntax error. (parser state: S6)
 (L:C = L1:C8)

My question where is my fault.

thanks
Wolfram

 

Dec 22, 2008 at 12:00 PM
Only to clarify:

I have this problem even with:
    TokenFilters.Add(new CodeOutlineFilter(false)); 
or
    TokenFilters.Add(new CodeOutlineFilter(true));
makes no diffenrence.

thanks
Wolfram


Coordinator
Dec 22, 2008 at 6:02 PM
My first advice in situations like these would be to enable tracing and to look at the token list, then parser trace and see when things mismatch
Dec 22, 2008 at 8:57 PM
that I did.

so I get the result that KOMMANDO is recognized than LF than some whitespace again LF than the this error. INDENT is not recognized.
KOMMANDO
 INDENT

Dec 22, 2008 at 9:06 PM
to add this:

that what is suprising me:
KOMMAND [Symbol]
[LF]
[LF]

is identified. But there are not two LF only one.

I hope TAB is correct for indent!?

regards Wolfram
Coordinator
Dec 22, 2008 at 9:53 PM
I see the problem. There's some inconsistency in the way NewLine symbol works together with outline token filter. Let me think how to fix it, I'll get back to you soon
Dec 22, 2008 at 10:11 PM
thanks!
Coordinator
Dec 23, 2008 at 6:14 PM
thank you for your review! about the bug, here's what you do. In CodeOutlineFilter.BeginFiltering method, add the following line at the beginning of the method:

context.Compiler.Grammar.WhitespaceChars =

" \t\r\n\v";

that would fix it. this is temporary workaround, will come up with permanent fix later

 

Dec 23, 2008 at 8:07 PM
Hi Roman,

thanks for the fix. But it seems not to work 100%

I added in the method:

 

public override IEnumerable<Token> BeginFiltering(CompilerContext context, IEnumerable<Token> tokens) {
    context.Compiler.Grammar.WhitespaceChars =
" \t\r\n\v";
.....

By using my sample:
KOMMANDO
    INDENT

The parser seems to get all tokens, but still a error ist there I tried to append Grammer.eof but same result.

Error:
Syntax error. (parser state: S8)
 (L:C = L1:C14)    

if I use this:
        indentzeile.Rule = Grammar.Indent + _ind + Grammar.NewLine+ Grammar.Eof ;
        Here the result:
                KOMMANDO [SYMBOL]
                [LF]
                [INDENT]   <<-- Here the ident is recognized with the patch you gave.
                INDENT     [Symbol]
                [LF]  <--- But there is no LF
                [DEDENT] <-- but there is no dedent   
        
Error:
            Syntax error. (parser state: S8)
             (L:C = L1:C14)    

or this:
            indentzeile.Rule = Grammar.Indent + _ind    ;
            Here the result:
                KOMMANDO [SYMBOL]
                [LF]
                [INDENT]   <<-- Here the ident is recognized with the patch you gave.
                INDENT     [Symbol]
                [LF]  <--- But there is no LF
            Error
                Syntax error. (parser state: S7)
                 (L:C = L1:C14)
        
            
    

 

Coordinator
Dec 23, 2008 at 8:22 PM
Edited Dec 23, 2008 at 8:23 PM
this is an expected behavior - CodeOutlineFilter on EOF "unindents" all previous indents and Scanner adds finishing end-of-line; it is expected that grammars use indentation in a "balanced" way; that makes it much easier to deal with unindents for languages like Python. So the reason is that you use incomplete test-like grammar. Just complete your grammar with unindent symbol that should complete the indented block (I guess).
Dec 23, 2008 at 8:30 PM
Dear Roman.
thanks
That was the problem

doing like that and it works.
indentzeile.Rule =
Grammar.Indent + _ind+ Grammar.NewLine+Grammar.Dedent+Grammar.Eof ;

merry christmas to you and your familie.

regards
Wolfram

 

Coordinator
Dec 23, 2008 at 9:54 PM
You're welcome and Merry Christmas!