Visual Studio Integration

Jun 14, 2012 at 9:30 AM


I want to  integrate my language into visual studio, i used lua as exemple.

I've got a problem with it. Using VsReadToken with the scanner.

I've got in my language a freetextliteral that bypass all stringliterals when browsing tokens.

so when in my language i've got "text", it returns a freetextliteral token instead of string.

When i parse the whole document, "text" is recognize as a string literal.

Any idea ?


Jun 14, 2012 at 5:04 PM

don't quite understand what's going on. Do you mean that Vs scan/parse works differently from "normal" parse? that can happen I guess, VS scan mode is quite limited, because it sees one line at a time. Try playing with terminal priorities.

Jun 14, 2012 at 7:46 PM

Yes that's it, it is the VS scan mode. And it is really different from normal parse.

FreeTextLiteral was already with low priority, i tried with high priority on string literals, but nothing has changed.

But, i think the problem is from my language, i'm not familiar to syntax/language processing, so i tried to use Irony. I have a lot of non terminals really few terminals. So maybe i'm doing something wrong, because when trying to set NonScanner flag on freetextliteral, i have only keyterms and identifiers, so it's really hard to determine what's currently parse.

But thanks for your reply

Jun 18, 2012 at 7:42 AM
Edited Jun 18, 2012 at 1:05 PM

I've got a new question. I'm using irony to parse whole document. The problem is irony doesn't care about tabulations, irony transforms tabulations with 4 white spaces, so parsing returns wrong locations. Is there a way to control that ?


EDIT 3 : I don't understand why irony doesn't build the parse tree when error is at the end of the file. Because it works pretty good when error is between correct statements

EDIT 2 : i noticed that it only appears when end of file. Working on it. So don't take care of EDIT1

EDIT 1  : few days ago, before i modified my grammar, irony grammar explorer returned an incomplete parse tree when there was an error (incomplete is normal because of the error), but now it doesn't return a tree. What can cause that behavior, or How can i tell irony that i want to get all correct statements and errors. Thx

Jun 26, 2012 at 5:56 AM

about tabs - there's a setting TabCount or smth like this, i think in ParsingContext, to change the "width" of tabs. You can also play with overwriting Grammar.SkipWhitespace method to treat tabs as non-whitespace or smth. All I can say - tabs are basically treated as "bunch of spaces"; there's a way to override it to some extent, you'll have to experiment with SkipWhitespace and similar methods

Apr 3, 2014 at 9:20 AM
Edited Apr 23, 2014 at 9:08 AM
I too am facing problem with Irony tab handling and Visual Studio.

In VS2012 it seems that they have eliminated the need to keep TabWidth "synchronized" with editors option. What I am essentially saying that TabWIdth=1 is always the right option, for >=VS2012. However, there is a little problem, Irony does not allow this.

Namely the line in SourceStream.cs:
      if (_tabWidth <= 1) 
        _tabWidth = 8;
change to
      if (_tabWidth < 1) 
        _tabWidth = 8;
and it will be okay(also TabWidth=1;). Though, I did not implement LanguageService, but "Classifier", which might or might not act the same as LanguageService with Spans. [Download Microsofdt VisualStudio 2010 SDK samples, and see IronPython.IsolatedShell) to see how the classifier is implemented.

To actually "tokenize" the shizzle,
        public IEnumerable<Token> Tokenize(string scriptLogic)
            var parseTree = _parser.Parse(scriptLogic);

            foreach (var token in parseTree.Tokens)
                yield return token;
and the GetClassificationSpans:
        IList<ClassificationSpan> IClassifier.GetClassificationSpans(SnapshotSpan span)
            var classifications = new List<ClassificationSpan>();

            int state = 0;
            int startIndex;
            int endIndex;

            // todo; perhap's we should split the line into tokens, and see what token do we have.
            foreach (var token in _codeService.Tokenize(span.GetText()))
                var startLine = token.Location.Line;
                var startOffset = token.Location.Column;
                var endOffset = startOffset + token.Length;

                var line = span.Snapshot.GetLineFromLineNumber(startLine + span.Start.GetContainingLine().LineNumber);

                startIndex = line.Start.Position + startOffset;
                endIndex = line.Start.Position + endOffset;

                if (endIndex > span.Snapshot.GetText().Length)
                    endIndex = span.Snapshot.GetText().Length;

                if (endIndex > startIndex)

                    var type = _classificationRegistryService.GetClassificationType(GetClassificationType(token));
                        new ClassificationSpan(new SnapshotSpan(span.Snapshot, startIndex, endIndex - startIndex), type));

            return classifications;