Using a Letter as an Operator

Sep 2, 2015 at 2:23 PM
I'm using Irony to implement a very basic calculator plus one unusual feature. The basic operations were simple enough using the built in BinaryOperationNode. Now I wanted to build in the unusual feature: I want the system to handle die rolls written out as nDm. E.G. Given the expression "1+2d6" the system simulates the rolling of two six-sided dice, and evaluates to the resulting number plus one.

Writing the node class was simple enough, and it works, provided I use a symbol as the operator (E.G. 2%6 instead of 2d6). If I try to use the character 'd' however I run into a "Invalid character: 'd'." error in the parse tree. I have tried setting the 'd' in RegisterOperators, MarkPunctuation, and an overwritten IsWhitespaceOrDelimiter method.

What am I missing? What steps do I need to take to fix this? There are no plans for identifiers or strings in this system, so breaking those features is not a concern, although if it can be done without breaking them it would of course be better, in case plans change later.
Sep 2, 2015 at 3:16 PM
Edited Sep 2, 2015 at 3:16 PM
Can you post your grammar file? It's easier to help you with it.
I suspect the problem is in the tokenization.
Sep 2, 2015 at 3:26 PM
public class ExpressionGrammar : Grammar {
        public ExpressionGrammar() : base(caseSensitive: false) {
            // 1. Terminals
            var number = new NumberLiteral("number");
            number.DefaultIntTypes = new TypeCode[] { TypeCode.Int32 };
            number.Options = NumberOptions.AllowSign | NumberOptions.AllowLetterAfter;

            // 2. Non-terminals
            var expression = new NonTerminal("Expression");
            var term = new NonTerminal("Term");
            var parentheticalExpression = 
                new NonTerminal("ParentheticalExpression");
            var binaryExpression =
                new NonTerminal("BinaryExpression", typeof(BinaryOperationNode));
            var binaryOperator = new NonTerminal("BinaryOperator", "operator");
            var rollExpression =
                new NonTerminal("RollExpression", typeof(RollOperationNode));
            var rollOperator = new NonTerminal("RollOperator", "operator");

            // 3. BNF rules
            expression.Rule = term | binaryExpression;
            term.Rule = number | parentheticalExpression;
            parentheticalExpression.Rule = "(" + expression + ")";
            binaryExpression.Rule = expression + binaryOperator + expression;
            binaryOperator.Rule = ToTerm("+") | "-" | "*" | "/";
            rollExpression.Rule = expression + rollOperator + expression;
            rollOperator.Rule = ToTerm("d");

            Root = expression;       // Set grammar root

            // 4. Operators precedence
            RegisterOperators(1, "+", "-");
            RegisterOperators(2, "*", "/");
            RegisterOperators(3, "d");

            // 5. Punctuation and transient terms
            MarkPunctuation("(", ")", "d");
            RegisterBracePair("(", ")");
            MarkTransient(term, expression, binaryOperator, parentheticalExpression, rollOperator);

            //6. Language flags. 
            // Automatically add NewLine before EOF so that our BNF rules work correctly when there's no final line break in source
            LanguageFlags = LanguageFlags.CreateAst;
        }

        public override void BuildAst(LanguageData language, ParseTree parseTree) {
            var opHandler = new OperatorHandler(language.Grammar.CaseSensitive);
            Util.Check(!parseTree.HasErrors(), "ParseTree has errors, cannot build AST.");
            var astContext = new InterpreterAstContext(language, opHandler);
            var astBuilder = new AstBuilder(astContext);
            astBuilder.BuildAst(parseTree);
        }

        public override bool IsWhitespaceOrDelimiter(char ch) {
            if (ch == 'd') return true;
            return base.IsWhitespaceOrDelimiter(ch);
        }

        public LanguageRuntime CreateRuntime(LanguageData language) {
            return new LanguageRuntime(language);
        }
    }
Sep 2, 2015 at 8:12 PM
Edited Sep 2, 2015 at 8:21 PM
Your rollexpression in not reachable from the start symbol, meaning it isn't a part of your grammar/language`. When I add it it works fine for me.

Your "d" is just another binary operator, so you can add it to that list.
Remove the override of isWhitespaceOrDelimiter, it's unneccesary. You also don't need to mark "d" as punctuation.

So either
expression.Rule = term | binaryExpression | rollExpression;
or
binaryOperator.Rule = ToTerm("+") | "-" | "*" | "/" | "d";

Sep 2, 2015 at 8:24 PM
Edited Sep 2, 2015 at 8:25 PM
I've made the changes you mentioned (whilst banging my head on my desk for missing the first bit...), but I am still getting the "Invalid character: 'd'." error.
public class ExpressionGrammar : Grammar {
        public ExpressionGrammar() : base(caseSensitive: false) {
            // 1. Terminals
            var number = new NumberLiteral("number");
            number.DefaultIntTypes = new TypeCode[] { TypeCode.Int32 };
            number.Options = NumberOptions.AllowSign | NumberOptions.AllowLetterAfter;

            // 2. Non-terminals
            var expression = new NonTerminal("Expression");
            var term = new NonTerminal("Term");
            var parentheticalExpression = 
                new NonTerminal("ParentheticalExpression");
            var binaryExpression =
                new NonTerminal("BinaryExpression", typeof(BinaryOperationNode));
            var binaryOperator = new NonTerminal("BinaryOperator", "operator");
            var rollExpression =
                new NonTerminal("RollExpression", typeof(RollOperationNode));
            var rollOperator = new NonTerminal("RollOperator", "operator");

            // 3. BNF rules
            expression.Rule = term | binaryExpression | rollExpression;
            term.Rule = number | parentheticalExpression;
            parentheticalExpression.Rule = "(" + expression + ")";
            binaryExpression.Rule = expression + binaryOperator + expression;
            binaryOperator.Rule = ToTerm("+") | "-" | "*" | "/";
            rollExpression.Rule = expression + rollOperator + expression;
            rollOperator.Rule = ToTerm("d");

            Root = expression;       // Set grammar root

            // 4. Operators precedence
            RegisterOperators(1, "+", "-");
            RegisterOperators(2, "*", "/");
            RegisterOperators(3, "d");

            // 5. Punctuation and transient terms
            MarkPunctuation("(", ")");
            RegisterBracePair("(", ")");
            MarkTransient(term, expression, binaryOperator, parentheticalExpression, rollOperator);

            //6. Language flags.
            LanguageFlags = LanguageFlags.CreateAst;
        }

        public override void BuildAst(LanguageData language, ParseTree parseTree) {
            var opHandler = new OperatorHandler(language.Grammar.CaseSensitive);
            Util.Check(!parseTree.HasErrors(), "ParseTree has errors, cannot build AST.");
            var astContext = new InterpreterAstContext(language, opHandler);
            var astBuilder = new AstBuilder(astContext);
            astBuilder.BuildAst(parseTree);
        }

        public LanguageRuntime CreateRuntime(LanguageData language) {
            return new LanguageRuntime(language);
        }
    }
Sep 2, 2015 at 8:45 PM
Edited Sep 2, 2015 at 11:57 PM
Your exact code (without the AST nodes since I don't have that code) works for me. Did you build again? (and refresh if you use grammar explorer)

edit: Sorry, seems like it doesn't work in all cases yet. I'll take a look at it tomorrow.
Sep 2, 2015 at 11:59 PM
You need to explicitly tell Irony that alphanumeric characters are allowed after the "d" keyword:
var rollOperatorTerminal = ToTerm("d");
rollOperatorTerminal.AllowAlphaAfterKeyword = true;

rollOperator.Rule = rollOperatorTerminal;
Sep 3, 2015 at 4:16 AM
That did it, thanks!