Include statement

Mar 11, 2010 at 12:40 PM

Hello.

Is it possible to imlement include statement? Like replace the statement with a text from external file while parsing.

And if possible, than how?

Coordinator
Mar 12, 2010 at 8:19 PM

The semantics of "include" might vary. One case (Python) it is a reference to external module that is executed at runtime - the module is parsed once and added to references of the current module. Implementation of this semantics is part of interpreter (with dedicated AST node), and will be part of interpreter implementation in some near future. 

The case you're probably talking about - including file as plain text at parse time. Not sure it is easy in current version, if possible at all. Are you sure you need this semantics, rather than referencing the external module? What is the language?

Roman

 

Mar 12, 2010 at 9:25 PM

The second case.

Language is BSDL, it is subset of VHDL. External package must be included in BSDL file as plain text and parsed all together. I am new to parsers and trying to port original BSDL grammar, see Standard_Use production here http://www.vhdl.org/vug_bbs/bsdl.parser .

Coordinator
Mar 15, 2010 at 5:36 AM

I don't have a bullet-proof solution, but I think you can implement this with custom token filter. You just catch "include" statement, then create a separate scanner, scan the external file and feed the tokens into the primary stream. See if you can make it work. I will get back to "include" implementations in some future and have a complete story.

Mar 18, 2010 at 10:34 AM

Thanks for the advice.
And thank you very much for the great tool.

Jun 3, 2010 at 9:01 PM

Has anyone picked this up yet?  Here's what I've got so far, but I'm missing something.  The pattern for my "include" statements are include "file path".

  1. I use a custom token filter to watch for an "include" token
  2. Pick up the file path after it which comes in the form of a string token
  3. Use that to open the file and read it
  4. Pass the text from that file to a new instance of the parser
  5. Return the tokens generated and try to add it to the stack of existing tokens

The problem is that a parse error will point to the beginning double quote in "file path".  Is there something I need to do around location?  Or is it something else?

 

using System.Collections.Generic;
using System.IO;

using Irony.Parsing;

namespace AJG.LanguageService.Parsing.TokenFilters
{
    public class IncludeTokenFilter : TokenFilter
    {
        TokenStack outputTokens = new TokenStack();
        bool previousTokenWasInclude;
        bool previousTokenWasIncludeFile;

        public override IEnumerable<Token> BeginFiltering(ParsingContext context, IEnumerable<Token> tokens)
        {
            Reset();

            foreach (Token token in tokens)
            {
                outputTokens.Push(token);

                if (token.Text == "include")
                {
                    previousTokenWasInclude = true;
                }
                else if (previousTokenWasInclude)
                {
                    TokenList includeTokens = ProcessIncludeFile(token.ValueString);

                    foreach (Token t in includeTokens)
                    {
                        outputTokens.Push(t);
                    }

                    previousTokenWasInclude = false;
                    previousTokenWasIncludeFile = true;
                }
                else if (previousTokenWasIncludeFile)
                {
                    previousTokenWasIncludeFile = false;
                }

                while (outputTokens.Count > 0)
                {
                    yield return outputTokens.Pop();
                }
            }
        }

        protected override void OnSetSourceLocation(SourceLocation location)
        {
            base.OnSetSourceLocation(location);
        }

        public override void Reset()
        {
            base.Reset();

            outputTokens.Clear();
        }

        private TokenList ProcessIncludeFile(string file)
        {
            try
            {
                StreamReader sr = new StreamReader(file);

                string source = sr.ReadToEnd();

                sr.Close();

                Parser parser = new Parser(new AJG.LanguageService.Grammar());

                ParseTree tree = parser.Parse(source);

                //remove EOF token before returning
                tree.Tokens.RemoveAt(tree.Tokens.Count - 1);

                return tree.Tokens;
            }
            catch
            {
                //return new instance to avoid defensive coding above for now...really need to return an error token
                return new TokenList();
            }
        }
    }
}

 

 

 

Jun 4, 2010 at 4:28 PM

Answered my own question here.  I messed around a little more with it and the following seems to work.  Two major differences: 1) I wasn't putting the new tokens onto the stack properly and 2) I now re-use the existing grammar instead of creating a new instance (I'm not sure how that makes a difference but it does).

using System.Collections.Generic;
using System.IO;

using Irony.Parsing;

namespace AJG.LanguageService.Parsing.TokenFilters
{
    public class IncludeTokenFilter : TokenFilter
    {
        TokenStack outputTokens = new TokenStack();
        bool previousTokenWasInclude;

        public override IEnumerable<Token> BeginFiltering(ParsingContext context, IEnumerable<Token> tokens)
        {
            Reset();

            foreach (Token token in tokens)
            {
                if (token.Text == "include")
                {
                    previousTokenWasInclude = true;
                }
                else if (previousTokenWasInclude)
                {
                    TokenList includeTokens = ProcessIncludeFile(context, token.ValueString);

                    for (int i = includeTokens.Count - 1; i >= 0; i--)
                    {
                        outputTokens.Push(includeTokens[i]);
                    }
                    
                    previousTokenWasInclude = false;
                }

                outputTokens.Push(token);

                while (outputTokens.Count > 0)
                {
                    yield return outputTokens.Pop();
                }
            }
        }

        protected override void OnSetSourceLocation(SourceLocation location)
        {
            base.OnSetSourceLocation(location);
        }

        public override void Reset()
        {
            base.Reset();

            outputTokens.Clear();
        }

        private TokenList ProcessIncludeFile(ParsingContext context, string file)
        {
            try
            {
                StreamReader sr = new StreamReader(file);

                string source = sr.ReadToEnd();

                sr.Close();
                sr.Dispose();

                Parser parser = new Parser(context.Language.Grammar);

                ParseTree tree = parser.Parse(source);

                //remove EOF token before returning
                tree.Tokens.RemoveAt(tree.Tokens.Count - 1);

                return tree.Tokens;
            }
            catch
            {
                //return new instance to avoid defensive coding above
                return new TokenList();
            }
        }
    }
}

Coordinator
Jun 4, 2010 at 5:03 PM

Hi

I will look at this and see maybe it is possible to create a generic support for include in core Irony - add this special filter with matching support in parser.

thanks

Roman