Priority between Identifiers and Keywords

Nov 2, 2014 at 12:27 PM

I am new to Irony, and I am impressed how easy it is to create a parser with it.

I created a parser for WIQL (Query Language for Microsoft TFS), this worked pretty good.

The reason for this is i want to extend the WIQL with e.g. taking the columns from another query.

For this I need some "special" Syntax which fits to the WIQL-Language, otherwise I cannot store my changes in TFS.

And now to the problem: I want to add my reference to the columns of another query like this:
SELECT ... FROM Workitems WHERE <normal where clause> AND [system.title] <> '%COLUMNSET%QueryPath%COLUMNSET%

And i want to assign this clause a special non-terminal.

My problem is that [system.title] <> '%COLUMNSET% is not regocnized, because [system.title] is taken as a fieldName.

The following grammer works perfectly:
      var selectStmt = new NonTerminal("selectStmt");
      this.Root = selectStmt;
      var refColumnSet = new NonTerminal("refColumnSet");
      refColumnSet.Rule = ToTerm("[system.title] <> '%COLUMNSET%");
      selectStmt.Rule = ToTerm("SELECT") + refColumnSet;
meaning, the refColumnSet is recognized.

But if a add the fieldname-definition like this:
  var selectStmt = new NonTerminal("selectStmt");
  this.Root = selectStmt;

  var fieldName = TerminalFactory.CreateSqlExtIdentifier(this, "fieldName"); //covers normal identifiers (abc) and quoted id's ([abc d], "abc d")

  var refColumnSet = new NonTerminal("refColumnSet");

  refColumnSet.Rule = ToTerm("[system.title] <> '%COLUMNSET%");
  fieldName.Priority = TerminalPriority.Low;

  selectStmt.Rule = ToTerm("SELECT") + fieldName | ToTerm("SELECT") + refColumnSet;

I only get the error "invalid character: '<', because before [system.title] is referenced as a fieldName. Decreasing the priority of fieldName does not help.

Any ideas how to solve this?

Nov 5, 2014 at 3:46 AM
I don't understand at all.. Using

ToTerm("[system.title] <> '%COLUMNSET%")

seems like completely wrong way to go. ToTerm converts its argument as a whole into a terminal, as a whole string literal. ToTerm should be only used when combining two literals with "+" operator, to avoid compiler confusing it with string concatenation operation:
so that
"a" + "b"
is interpreted by c# as "ab", before even Irony has a chance to look at it; instead you write

ToTerm("a") + "b"

then Irony will correctly understand that this means literal "a" followed by some spaces followed by "b".
That's the only reason to use ToTerm() method!!!
So try to express the rules without it. It does not matter what errors it gives you currently, the arrangement is wrong to begin with
Nov 5, 2014 at 7:46 AM
Hi Roman,

Thanks for your answer - probably my explanation was too short. I hope the below is not too short again. (Or I am completely wrong …).

I have tried to give a very short example. Below is a little bit more.

I have “normal” where-conditions (like “a > 15”). I treat these conditions as it should be with the non-Terminals fieldname, operator and value. This works perfectly.

But then I have a “special” where-condition like “([system.title] <> '%COLUMNSET% /path/QueryName’)”.

This is of course again a combination of non-terminals fieldname, operator and value.

But I would like to treat it differently, because this syntax has a special meaning, and I can find it much easier in the parse tree if it is not fieldname, operator and value, but something like
Columnsetref + querypath. Therefore I really want to treat “([system.title] <> '%COLUMNSET%)” as one string.

Unfortunately I cannot define a special syntax for this columnSet, because I have to store my query definition in TFS, and this is only possible with a syntax TFS understands. Therefore I created this “workaround”.

Above solution does not work, because irony detects it (correctly) as fieldname, operator and value. Therefore I would like to give the Columnsetref + querypath higher priority than fieldname, operator and value. Is this possible?

I hope it is more clear now.

Nov 6, 2014 at 12:25 PM
At least for the problem in the "very small" grammar I found a solution.

I changed
var fieldName = TerminalFactory.CreateSqlExtIdentifier(this, "fieldName");
var fieldName = new IdentifierTerminal("fieldName", "[].", "[");
With this change, my strange "[system.title] <> '%COLUMNSET%" is recognized as own token.

But this is just a workaround, is it possible to solve this with the "correct" fieldname definition for the SQL identifier?

Nov 12, 2014 at 6:25 PM
I don't think this workaround would really work; how about spaces inside? Identifier terminal would stop at space char thinking this is the end.
I don't quite understand, in pure human terms, what would be the criteria for recognizing this special fragment as one special token (fieldname)? Like if we "hit the string that starts with '[' and ends with '%' THEN this is one special fieldName token. Formulate this rule in pure human language, then try to find an implementation (pre-existing terminal in Irony) or create custom terminal that can handle these special strings
Nov 16, 2014 at 11:11 AM
I have already recognized (and solved) the problem with the spaces with an own "TerminalIdentifierWithSpaces".

Regarding your question: The criteria for the special fragment are just is exactly "[system.title] <> '%COLUMNSET%". (This is really a constant).

But my main question is still open: Why is "[system.title] <> '%COLUMNSET%" recognized as own token with my own definition of fieldName (new IdentifierTerminal("fieldName", "[].", "[")), but it is not with fieldName = TerminalFactory.CreateSqlExtIdentifier(this, "fieldName");?

Nov 20, 2014 at 6:07 AM
because CreateSqlExtIdentifier defines identifier with optionally quoted by square brackets; so right bracket is a token-end char for terminal; the other version allows brackets INSIDE terminal