MarkTransient nodes still appearing in parse tree / getting adding to AST

Jun 10, 2013 at 11:03 PM
Edited Jun 10, 2013 at 11:04 PM

I am having an issue where nodes marked as transient are still showing up in the parse tree. In my case, the node getting added this way is 'expression'. They are also getting added to the AST, but as null nodes. There is no AstNodeType set for 'expression.'

Instead of a parse tree like:
      binaryExpression 2 + 6
      numberLiteral 6
I'm getting:
         binaryExpression 2 + 6
      numberLiteral 6
Sorry to codedump, but I'm not sure what other relevant information I could provide, so here is my grammar:
// Terminals
var blockComment = new CommentTerminal("DelimitedComment", "{", "}");
MarkPunctuation(",", "(", ")");

var numberLiteral = new NumberLiteral("numberLiteral", NumberOptions.AllowSign | NumberOptions.AllowStartEndDot | NumberOptions.AllowUnderscore, typeof(NumberNode))
    DefaultFloatType = TypeCode.Decimal

var measurementUnit = new IdentifierTerminal("measurementUnit")
    AllFirstChars = LETTERS,
    AllChars = LETTERS

var sensor = new IdentifierTerminal("sensor")
    AllFirstChars = "$",
    AllChars = LETTERS + DIGITS

var SUM = ToTerm("sum", "sum");
var MEAN = ToTerm("mean", "mean");
var MIN = ToTerm("min", "min");
var MAX = ToTerm("max", "max");

var LPAREN = ToTerm("(", "lparen");
var RPAREN = ToTerm(")", "rparen");
var COMMA = ToTerm(",", "comma");

// Nonterminals
var result = new NonTerminal("result", typeof(ResultNode));

var sensorRead = new NonTerminal("sensorRead", typeof(SensorNode));
var qualifiedNumber = new NonTerminal("qualifiedNumber", typeof(QualifiedNumberNode));
var qualifiedPrimaryExpression = new NonTerminal("qualifiedPrimaryExpression");
var qualifiedExpression = new NonTerminal("qualifiedExpression");
var qualifiedBinaryExpression = new NonTerminal("qualifiedBinaryExpression", typeof(QualifiedBinOpNode));
var qualifiedAggregateFunctionApplication = new NonTerminal("qualifiedAggregateFunctionApplication", typeof(AggregateFunctionInvocationNode));

var qualifiedArgument = new NonTerminal("qualifiedArgument");
var qualifiedArgumentList = new NonTerminal("qualifiedArgumentList", typeof(QualifiedArgumentList));

var primaryExpression = new NonTerminal("primaryExpression");
var expression = new NonTerminal("expression");
var binaryExpression = new NonTerminal("binaryExpression", typeof(BinOpNode));

var infix = new NonTerminal("infix");
var additive = new NonTerminal("additive");
var multiplicative = new NonTerminal("multiplicative");
var aggregateFunction = new NonTerminal("aggregateFunction");

// Rules
// Operators
aggregateFunction.Rule = SUM | MEAN | MIN | MAX;

additive.Rule = ToTerm("+") | "-";
multiplicative.Rule = ToTerm("*") | "/" | "^";

// Measurement-qualified
result.Rule = Empty;
result.Rule |= qualifiedExpression;

sensorRead.Rule = sensor;

qualifiedNumber.Rule = primaryExpression + measurementUnit;

qualifiedPrimaryExpression.Rule = sensorRead;
qualifiedPrimaryExpression.Rule = qualifiedNumber;
qualifiedPrimaryExpression.Rule |= LPAREN + qualifiedExpression + RPAREN;

qualifiedBinaryExpression.Rule = qualifiedExpression + additive + qualifiedExpression;
qualifiedBinaryExpression.Rule |= ImplyPrecedenceHere(25) + primaryExpression + multiplicative + qualifiedExpression;
qualifiedBinaryExpression.Rule |= ImplyPrecedenceHere(20) + qualifiedExpression + multiplicative + primaryExpression;

qualifiedAggregateFunctionApplication.Rule = aggregateFunction + LPAREN + qualifiedArgumentList + RPAREN;

qualifiedExpression.Rule = qualifiedPrimaryExpression | qualifiedBinaryExpression | qualifiedAggregateFunctionApplication;

qualifiedArgument.Rule = qualifiedExpression;

qualifiedArgumentList.Rule = MakeStarRule(qualifiedArgumentList, COMMA, qualifiedArgument);

primaryExpression.Rule = numberLiteral;
primaryExpression.Rule |= LPAREN + expression + RPAREN;

binaryExpression.Rule = expression + additive + expression;
binaryExpression.Rule |= expression + multiplicative + expression;

expression.Rule = binaryExpression | primaryExpression;

// Operator Precedence
RegisterOperators(5, Associativity.Left, "+", "-");
RegisterOperators(10, Associativity.Left, "*", "/");
RegisterOperators(15, Associativity.Right, "^"); // we'll worry about this later.

Root = result;

Measurement units are user-configurable and not available at parse time; the idea is that a user could enter "20 Gallons + 10 Liters" and get a reasonable result. I was trying to fix an issue where expressions like "(2 * 6) Gallons" were not parsing; they are now parsing but not in the way that I expect. I'm new to BNF so please excuse any inelegance.

Any help would be greatly appreciated.

Thank you,
Jun 11, 2013 at 7:08 AM
your first version (that I got in initial notification email) had 'ReduceHere()' hint in the Rule for expression, so that was my initial guess that this extra hint might cause this behavior. In the current edited version the hint is gone. Does it still work incorrectly? And I guess it does work for other nonterminals like 'additive' and 'multiplicative'. Can you just look at parsing log and to to see what's going on there? If it does not give a clue, try to stop in debugger in place where the parser analyses this Transient flag and 'pops' the child and uses it instead of parent? Let me know how it goes; if you can't find it, I'll go step by step myself. sorry just swamped with other things currently
Jun 11, 2013 at 2:17 PM
Edited Jun 11, 2013 at 2:33 PM

I am still having the general issue with posted version, but that particular expression isn't causing the same issue now, sorry. (1*2+6) Gallons is an expression that is currently resulting in surviving 'expression' nodes. The only transient node that I am having the issue with is 'expression.' Looking through the parse log, I wasn't able to find any case where it was using it instead of the parent, but I could also be reading the log wrong. I'll try to step through it in the debugger soon to see if I can figure anything else out.

Thank you for the quick reply!
  • Chris