Creating intellisense

Apr 14, 2010 at 7:00 AM


First off, thanks for making such a useful library. I am writing my first DSL using it.

I have a question regarding intellisense / autocomplete.

I would like to implement autocomplete in a way that, depending on the context, correct options are shown.

For e.g. if the grammar allows to write 10 + 5, i.e. a NumberLiteral + [+|-|*|/] + NumberLiteral

When the first NumberLiteral is typed and a space is pressed I what to show a autocomplete dropdown with + , - , * , /  in it.

For this I need to know what NonTerminal(s) is the parser expecting / will allow after the NumberLiteral.

If there a way to extract this information?

... As DSL is a new things for me and I am not able to use the right vocabulary to express my problem better.


Apr 14, 2010 at 1:39 PM

Well, in general, to support intellisense you need to have a parser that constantly re-parses the source text (on background thread). When parser reaches the current position (carret position), you can retrieve the currently expected set of terms by looking at current parser state's ExpectedTerms property.

Apr 14, 2010 at 6:59 PM


Great response to a great question!  I've always wondered how difficult it would be to implement this and you make it sound pretty straight forward. 

I would assume that the best way to implement this behavior is to add an event handler to your UI control that would fire when onKeyPressed occurs.  Then the action for the event handler would first determine the ExpectedTerms based off of the character after the last Terminal or Non-Terminal depending on whether or not the last node was a Terminal or Non-Terminal. Then, depending on the interaction of the control, return the list for a drop down or populate your initial control with the remaining characters of the first ExpectedTerms.

I may have to play around with this in the grammar explorer.  I think this would be something great to show everyone how powerful, yet simplistic, the Irony project is.


Apr 14, 2010 at 7:46 PM

Thanks Roman. 

Although I must confess, I have no idea how this has to be implemented, at least I know where to begin. Will update the thread if I could get this to work.

MindCore, in case you make any progress, please share some ideas.

Apr 14, 2010 at 8:18 PM

Playing a bit more, I see what you mean. Adding this code to the end of EditorViewAdaptor.UpdateParsedSource, gave me the expected tokens at the end of the source.

        if (newTree.ParserMessages.Count > 0)
            foreach (var msg in newTree.ParserMessages)
                foreach (var terminal in msg.ParserState.ExpectedTerminals)

Now I need to only Parse the source until the current caret position, and show the suggestion based on parsing the ExpectedTerminals.

So what is left is glue code and not Irony related.

I am correct in my understanding?

Apr 15, 2010 at 5:28 AM

thanks for jumping in, Mindcore! Really appreciate any help answering other people's questions. I'm not always available, so the question may stay unanswered for hours and days.

hitezh, about your code. I don't think this is quite right approach. newTree.ParserMessages is a collection of parsing errors - from entire file. First, it won't show anything if somehow at the moment the code is correct. Second, if there are syntax errors in several places, you'll collect all these places, get parser states and bundle all expected symbols together from all places. That's much more than you might actually expect in CURRENT CURSOR position. 

So what you need to do is catch the moment when background parser reaches current edit position. I think you can hook into TokenCreated event, and watch the token position. Once you hit current position, you retrieve parser state and remember it somewhere. Now usually intellisense is not invoked on any keystroke in any position. In VS integration package, you specify certain chars that fire drop-down - usually it is "." (dot), or space. That's when you start to check the parser state in current position that you saved from background parse, and decide what to show in dropdown.

This is just a sketch, but gives some initial direction



Apr 15, 2010 at 5:45 AM

PS Mindcore - if you can cook something like this for Grammar Explorer (what you described) - I'm all for it, and would be happy to add it



Apr 15, 2010 at 5:59 AM
Edited Apr 15, 2010 at 6:00 AM

Mindcore- Our goal is also to achieve intellisense by parsing in web application. I will be more than happy if you can share your thoughts.

Apr 15, 2010 at 8:29 PM

We are also interested in creating intellisense for use in a data editor for our application (for simple expressions in any of four grammars).  I will be checking back on this thread for ideas (and hopefully to contribute if I have time!).

Apr 16, 2010 at 9:33 PM

Hi Roman,

Thanks for the direction. I missed the point, since my language can have only 1 statement per file!

I think I know know where to get the data. As suggested I am catching the TokenCreated Event and when


+ (e.Context.Source).Location {(1:14)} Irony.Parsing.SourceLocation
+ e.Context.CurrentParserState.ExpectedTerminals Count = 2 Irony.Parsing.TerminalSet

e.Context.Source.Location >= current_caret_location

I can use the save the e.Context.CurrentParserState.ExpectedTerminals and use them as needed.

Does that sound right? 



Apr 16, 2010 at 9:53 PM

Sounds right. Good luck


Apr 16, 2010 at 9:55 PM

Hey hitezh,

Be careful on the comparison of the Location properties.  There is no direct way to do a greater than or equal to against two locations unless you add it to the SourceLocation class.  This class has two properties that identify the relevant location values for the line and the column.  This can be a little tricky if you grammar supports terminals that span multiple lines.

Also, your approach is sound. I believe your next step would be to set up a background thread with it's own ParsingContext.  This ParsingContext is going to be the object you tie your TokenCreated event to.  I've started something similar and have code in place to get the CaretLocation on the GrammarExplorer.  Hopefully I'll have something soon.



Do you have any suggestions on making this as efficient as possible?  It appears that the background parser would have to parse the entire text up until the token before the CaretLocation.  I see this being a performance issue as a file grows larger and larger.  I'm still trying to work through the logic but I do have much of the UI in the Grammar Explorer done; I just have to tie it into my AutoComplete subclass.



Apr 16, 2010 at 10:29 PM

I suggest not to add an extra parser on extra thread, but to use the process that colorizes tokens, which is already there. Just add the code that saves current parser state when parser goes thru current location, so parser state will be available to intellisense code


Apr 17, 2010 at 6:02 AM
mindcore wrote:

This can be a little tricky if you grammar supports terminals that span multiple lines.

Multiline grammars are the problem, Else it is easy enough to just send the text of the current line up to the caret location to the parser, thus improving the performance. For added performance, The call to parser should happen only on a specific keypress, like dot or space or ctrl+space. Meanwhile, my experiments were using the parser from the colorizer. I am hooking the event to that parser.