Linescanner provides false line numbers - bug?

Nov 9, 2012 at 12:38 PM
Edited Nov 9, 2012 at 12:59 PM


According to the provided examples, ScanTokenAndProvideInfoAboutIt() calls _parser.Scanner.VsReadToken(ref state) to retrieve the next token. Basically, everythings works fine except for token.Location.line. This field seems to be steadily incremented, rather than containing the actual line number of the token. Eg. for the very first token in the file Location.line==2 the first time, then 192, then 195, 213 etc etc

Doing a quick review of the code, I found method VsSetSource() in Scanner.cs, which is called by IScanner.SetSource(). I guess the problem is that one a Context.Source is available, the line-number for newLoc is incremented every time. But since VsSetSource()is not called in succession for every line (but rather for lines currently modified), the line-numbers will - except for the very first time - never be correct.

The question is: What to do about it? IScanner.SetSource() does not provide any information about which line is currently processed, it just passes the current line as a string... Any ideas?



Nov 9, 2012 at 5:56 PM

Honestly, have no idea how to fix this. Did not touch this VS integration interface for some time. It would be great if you investigate and suggest a solution or workaround?



Nov 9, 2012 at 7:19 PM
Edited Nov 21, 2012 at 3:46 PM

Hi again!

I invested a little more time and came up with a solution that does the trick. I don't know if this would be the preferred solution from Microsoft, but I couldn't find another way, and the solution is relatively simple. In the following paragraphs I assume that whoever reads this is familiar with the basic setup / classes of a Visual Studio Language Package, to keep it simple. If something is not clear (or I made some mistake), feel free to say so and I will try to explain better (or correct).

(1) In the main LanguageService class, override the GetColorizer Method:

class MyLanguageService : LanguageService {
    public override Colorizer GetColorizer(IVsTextLines buffer) {
        return new MyColorizer(this, buffer, GetScanner(buffer));

(2) Clearly, we now need to provide a costum colorizer class, so create a new MyColorizer.cs file:

class MyColorizer : Colorizer {
    public MyColorizer(LanguageService svc, IVsTextLines buffer, IScanner scanner) 
: base(svc, buffer, scanner) { } // we need some way to tell the scanner the correct line, otherwise the location // cannot be correctly determined, and thus the scope cannot be determined... public override TokenInfo[] GetLineInfo(IVsTextLines buffer,
IVsTextColorState colorState) { if (Scanner != null) { ((MyLineScanner)Scanner).SetSourceLine(line); } return base.GetLineInfo(buffer, line, colorState); } }

Edit/Note: You must do the same thing with ColorizeLine(...). I'm too lazy to edit the code above :-)

(3) Now comes the part that I don't like very much, because it's kind of a hack, but whatever... We need to insert the SetSourceLine() method into the LineScanner class, and adapt the SetSource() method accordingly

class MyLineScanner: IScanner {
    private int _line;
    private Parser _parser;

    public MyLineScanner(IVsTextLines buffer) {
        _buffer = buffer;
        _parser = new Parser(new MyGrammar());
        _parser.Context.Mode = ParseMode.VsLineScan;

    public void SetSourceLine(int line) {
        _line = line;

    void IScanner.SetSource(string source, int offset) {
        _parser.Scanner.VsSetSource(source, offset, _line);

(4) Finally, we need to extend the interface of the VsSetSource() method to take an additional line parameter, in file Scanner.cs

public class Scanner  {
    public void VsSetSource(string text, int offset, int cur_line) {
        var line = cur_line;// Context.Source == null ? 0 : Context.Source.Location.Line;
        var newLoc = new SourceLocation(offset, line, 0);
        Context.Source = new SourceStream(text, Context.Language.Grammar.CaseSensitive, Context.TabWidth, newLoc); 

That's basically it. Until now, this solution works perfectly fine for me. I don't think it has any negative side effects, because usually, the LineScanner doesn't need the Location.Line information anyway (I guess otherwise someone would've reported the problem earlier), and the Line parameter was definitely NOT correct...