Delimiting statements by newline

Aug 26, 2010 at 5:46 AM
Edited Aug 26, 2010 at 6:42 AM

Hello all,

I have run into a slight problem.  I have build a grammar that allows for the following two types of statements (simplified):

OptionalBracketSection.Rule = ("[" + Expression + "]") | Empty;
TypeStatement.Rule = TypeKeyword + Identifier + OptionalSquareBracketSection;


LoopStatement.Rule = "[" + Expression + "]" + Identifier + "(" + Identifier + ")";


Now...the problem is this.  If have the following:

Type blah

[5] ( Test )


The grammar treats it as 'Type blah[5]' and chokes on the '( Test )' - instead of using 'Empty' in the OptionalBracketSection rule.  I didn't get any shift-reduce conflicts which scares me (how many more such mess-ups have I made that I don't know about?).

Is there a way to fix this?  These statements are wrapped in in a StatementList that uses MakeStarRule, but when I tried to set the delimiter in that to '\n' it didn't seem to fix the problem - surely if I set '\n' as the delimiter it should force the end of a statement when it encounters one?

Many thanks!


edit: Tried adding + NewLine to the end of TypeStatement.Rule, but this causes everything else to keel over.

Aug 26, 2010 at 6:35 AM
Edited Aug 26, 2010 at 6:41 AM

[sorry double post]

Aug 26, 2010 at 4:01 PM

So, did this help - using NewLine as a statement separator instead of "\n"?


Aug 27, 2010 at 12:34 AM

No, it didn't.  If caused newlines to stop working in other rules as before (as whitespace).

I also get a problem with the fact that Expression can be

(Identifier + "[" + Expression + "]") | Identifier


Which causes [test[test]] ( Test ) to work incorrectly, as the parser matches the inner [test] part to a LoopStatement.

I might end up having to change the language a bit and make LoopStatement the only thing that uses square brackets - which is an option.  I really don't know enough about parser theory to know if the grammar should be able to 'roll back' its parsing stack indefinitely if it can't find a suitable token, in order to find something else that matches.

I assume this is not the case?



Aug 27, 2010 at 4:15 PM

You probably do not need to change the language but reorganize the grammar. Pay attention to Grammar Errors page - it tells you about conflicts in your grammar - cases when parser has several ways of interpreting the input. No, parser cannot "backtrack". I may introduce it in the future, but for now it is not there. 

On the other hand, adjusting the language (if it is under your control) might be a good idea. to make syntax less ambiguous for users and for the parser. Majority of existing languages are unambiguous enough to be parsed by LR-based parsers.


Aug 28, 2010 at 7:35 AM

Yeah, I think I'll adjust the language.  There were no grammars, unfortunately - perhaps the fact that there were no errors from a syntax that is ambigous is something that needs to be logged as a bug?  I'm happy to try to put together a small sample that exhibits this behaviour and put in an issue.


Aug 29, 2010 at 11:36 AM

There's definitely something screwy going on; for the text

if name <> "*col.m52" then
    interlaced: 4

and the rules


JumpIfFalse.Rule = Empty;
JumpHere.Rule = Empty;
IfStatement.Rule = ToTerm("if") + BoolExpression + JumpIfFalse + "then" + "(" + InnerStatements + ")" + (JumpHere | ElseSection);
ElseSection.Rule = ToTerm("else") + (("(" + InnerStatements + ")") | IfStatement); 

The 'JumpHere' node is never processed.  There is no way it can match the 'else'...

Sep 7, 2010 at 4:41 PM

Sorry for the delay in response, was on vacation for a week. 

I don't quite understand how your grammar supposed to work. Please make it more clear - what does it mean "never processed"?

If parser construction does not give any errors, then try to Trace the parser (run with Parser Trace checkbox checked), and try to follow the shift-reduce steps and see why it fails. 


Sep 9, 2010 at 12:51 AM

Tracked the issue down, and it was answered in the other post - I didn't have a default node type set up and AST nodes weren't being created.  Many thanks!