How to implement line continuation?

Apr 6, 2013 at 10:45 PM
The PowerShell line continuation rules look like this:
Unlike some popular languages, PowerShell does not consider line-terminator characters (§2.2.2) to be white space. However, a line terminator can be treated as white space by preceding it immediately by a backtick character, ` (U+0060). This is necessary when the contents of a line are complete syntactically, yet the following line contains tokens intended to be associated with the previous line.
I tried implementing this behavior by overriding SkipWhitespace. If source.PreviewChar is backtick (``), and source.NextPreviewChar is \r or \n, then increment source.PreviewPosition.

That works unless the backtick is followed by \r\n. PowerShell says that those 3 characters together form one whitespace, but there is no NextNextPreviewChar I can examine. I tried modifying my local copy of Irony to provide such a method, and that functioned as I expected. @rivanstov, you could merge that change in to Irony and it would work for me, but maybe you think there's a better way to deal with this.

I took a look at Irony.Parsing.LineContinuationTerminal, but it wasn't clear to me how it should work, and there are no docs or comments.

Any recommendations?
Apr 6, 2013 at 11:39 PM
I think LineContinuationTerminal should work here. Look in LineContinuationTests.cs in unit tests, and it is actually used in GwBasic grammar.
Apr 7, 2013 at 7:02 AM
I figured it out, and it was straightforward, so I'm writing it here if anyone tries to do something similar.

There were two changes I made that are specific to my code, which most people will ignore:
  • I had overloaded SkipWhitespace to handle line continuation, and I removed that code.
  • I had a very greedy token that was eating the line continuation character, so I fixed that.
So, to implement line continuation for the PowerShell grammar:
  1. Create an instance of Irony.Parsing.LineContinuationTerminal, passing the line continuation character(s) to the constructor
  2. Set LineContinuationTerminal.LineTerminators to "\r\n".
  3. NonGrammarTerminals.Add with your LineContinuationTerminal.
Note that Irony has a special case for line continuation with \r\n together.

Because I like creating new classes, I made one for this purpose:
    // A non-grammar terminal that controls how the parser handles line continuation.
    class BacktickLineContinuationTerminal : LineContinuationTerminal
        public BacktickLineContinuationTerminal()
            : base(typeof(BacktickLineContinuationTerminal).Name, "`")
            this.LineTerminators = "\r\n";
NonGrammarTerminals.Add(new BacktickLineContinuationTerminal());
(I didn't bother saving the object in a variable, since I don't need to refer to it again.)

If you're interested in code diffs: