Trouble with whitespace-delimited list

Jul 23, 2009 at 3:49 PM
I'm having trouble making my grammar parse a whitespace-delimited list of numbers and identifiers.  For example:

class TestGrammar : Grammar { public TestGrammar() { var number = new NumberLiteral("number"); var identifier = new IdentifierTerminal("identifier"); var numOrId = new NonTerminal("numOrId"); numOrId.Rule = MakePlusRule(numOrId, null, number | identifier); this.Root = numOrId; } }

For some reason, this grammar accepts "5a" as valid.  How can I make whitespace required between the number and identifier?

Thanks for your help.

Jul 23, 2009 at 5:46 PM

That's a tough one, surprisingly. Essentially, all the components are doing its job properly: NumberLiteral stops scanning when it sees the letter that follows the number and returns number token to scanner. The scanner continues, picks up Identifier literal, which also returns a valid token, and everybody is happy. Except, from common sense prospective, that should be rejected. The proper way to fix it is to add extra check to number literal to report error if the number is followed by a letter. We've run into this before, and planned to add some functionality to support this rejection, just still didn't get to this yet. I will try to add this in the next code drop, I'm doing several scanner/terminals enhancements currently, right in this area. Until then - I hope you can live with that for now...

Just to make it clear, it will happen in latest code drops in Sources page, not in old "released" version in downloads. I hope you're using this version.


Jul 23, 2009 at 5:50 PM

Can you give me any advice on how to "add an extra check to number literal to report error if the number is followed by a letter"?  I'm a beginner at Irony.

In my full grammar, there are certain letters which are acceptable numeric suffixes.  For example, "5deg" represents 5 degrees.  I suppose the non-alpha character check would have to happen after it scanned past  the suffix?

Thanks again for your help.

Jul 23, 2009 at 6:04 PM

Are you adding these valid suffixes explicitly to number literal? you should do it.

As for workaround, you can change or inherit the number literal and add validation at the end (when it produces token). If the following char is a letter, return null instead of token.


Jul 23, 2009 at 6:09 PM

I'm using AddSuffixCodes() to add my suffixes.  I got it working.  Here's the code I used in case anyone else needs this:

class CustomNumberLiteral : NumberLiteral
	public CustomNumberLiteral(string name) : base(name)
	protected override Token CreateToken(CompilerContext context, ISourceStream source, CompoundTokenDetails details)
		Token token = base.CreateToken(context, source, details);
		return char.IsLetter(source.CurrentChar) ? null : token;

Jul 23, 2009 at 6:13 PM

Yea, that looks right. Congrats - you're no longer newbie in Irony anymore!

Aug 12, 2009 at 1:27 PM

In latest drop, there's an extra flag in NumberLiteral that allows detecting this situation automatically and reports an error