irony Wiki & Documentation Rss Feed

New Comment on "Expression Grammar sample"

homaica — Tue, 05 Feb 2013 14:15:36 GMT

Hello Can someone explain me the purpose of MarkTransient ? Thank you

New Comment on "ContribProjects"

pgeerkens — Sat, 13 Oct 2012 20:37:02 GMT

re: P5: Handling implied "+" between string literals This seems more naturally a grammatical construct caught by the Parser than a lexical one caught by the Scanner. Is there a particular reason why that is incorrect? Pieter

Updated Wiki: Home

rivantsov — Fri, 16 Mar 2012 19:41:47 GMT

Irony is a development kit for implementing languages on .NET platform. Unlike most existing yacc/lex-style solutions Irony does not employ any scanner or parser code generation from grammar specifications written in a specialized meta-language. In Irony the target language grammar is coded directly in c# using operator overloading to express grammar constructs. Irony's scanner and parser modules use the grammar encoded as c# class to control the parsing process. See the expression grammar sample for an example of grammar definition in c# class, and using it in a working parser.

Download contents
The download zip contains core Irony libraries implementing a parsing engine, an interpreter, a number of sample grammars, unit test projects, and Grammar Explorer tool for viewing and debugging your languages (see picture below).
We provide multiple sample grammars for languages like GW Basic, Java, c#, Scheme, SQL, JSON and some others. Irony includes a ready-to-use expression evaluator that can be easily plugged-in into a .NET application.

System Requirements
Windows 7, .NET Framework 4.0, Visual Studio 2010

Irony on the Web
Irony presentation at LangNET 2009 symposium
Scott Hanselman about Irony.
Writing your first Domain-Specific Language
A Google-like Full Text Search engine based on Irony parser.
Writing your first Visual Studio Language Service
Writing a calculator in c# using Irony

More information
Expression Grammar sample
Contributors
Willing to Contribute?
Demo Running Instructions
Wikibook: Irony - Language Implementation Kit - everybody is welcome to contribute.

My blog and other projects
Irony blog
VITA Application Framework

Updated Wiki: Home

rivantsov — Thu, 05 Jan 2012 09:49:12 GMT

New Comment on "Demo Running Instructions"

tmaisnam — Tue, 22 Nov 2011 07:08:49 GMT

Fabulous tool ! breezed through installation too. thanks !!

Updated Wiki: Home

rivantsov — Wed, 19 Oct 2011 16:33:23 GMT

Updated Wiki: Home

rivantsov — Wed, 19 Oct 2011 16:27:27 GMT

Irony is a development kit for implementing languages on .NET platform. Unlike most existing yacc/lex-style solutions Irony does not employ any scanner or parser code generation from grammar specifications written in a specialized meta-language. In Irony the target language grammar is coded directly in c# using operator overloading to express grammar constructs. Irony's scanner and parser modules use the grammar encoded as c# class to control the parsing process. See the expression grammar sample for an example of grammar definition in c# class, and using it in a working parser.

Download contents
The download zip contains core Irony libraries implementing a parsing engine, an interpreter, a number of sample grammars, unit test projects, and Grammar Explorer tool for viewing and debugging your languages (see picture below).
We provide multiple sample grammars for languages like GW Basic, Java, c#, Scheme, SQL, JSON and some others. Irony includes a ready-to-use expression evaluator that can be easily plugged-in into a .NET application.

System Requirements
Windows 7, .NET Framework 4.0, Visual Studio 2010

Irony on the Web
Irony presentation at LangNET 2009 symposium
Scott Hanselman about Irony.
Writing your first Domain-Specific Language
A Google-like Full Text Search engine based on Irony parser.
Irony presentation at 2009 Lang.NET symposium
Writing your first Visual Studio Language Service
Writing a calculator in c# using Irony

More information
Expression Grammar sample
Contributors
Willing to Contribute?
Demo Running Instructions
Wikibook: Irony - Language Implementation Kit - everybody is welcome to contribute.

My blog and other projects
Irony blog
VITA - ORM done right

Updated Wiki: Home

rivantsov — Wed, 19 Oct 2011 16:26:43 GMT

02/10/2009 - new article on CodeProject: Writing your first Visual Studio Language Service by Ben Morrison about using Irony for language integration into Visual Studio.
April 20, 2009 - Recordings of presentations at 2009 Lang.NET symposium are available online. I've made a couple of presentations there, the first one about Irony. Watch it, come back with comments. Watch other recordings - very interesting stuff!

Irony on the Web
Irony presentation at LangNET 2009 symposium
Scott Hanselman about Irony.
Writing your first Domain-Specific Language
A Google-like Full Text Search engine based on Irony parser.
Irony presentation at 2009 Lang.NET symposium
Writing your first Visual Studio Language Service
Writing a calculator in c# using Irony

More information
Expression Grammar sample
Contributors
Willing to Contribute?
Demo Running Instructions
Wikibook: Irony - Language Implementation Kit - everybody is welcome to contribute.

My blog and other projects
Irony blog
VITA - ORM done right

Updated Wiki: Home

rivantsov — Wed, 19 Oct 2011 16:25:58 GMT

02/10/2009 - new article on CodeProject: Writing your first Visual Studio Language Service by Ben Morrison about using Irony for language integration into Visual Studio.
April 20, 2009 - Recordings of presentations at 2009 Lang.NET symposium are available online. I've made a couple of presentations there, the first one about Irony. Watch it, come back with comments. Watch other recordings - very interesting stuff!

Updated Wiki: Home

rivantsov — Wed, 19 Oct 2011 16:20:53 GMT

02/10/2009 - new article on CodeProject: Writing your first Visual Studio Language Service by Ben Morrison about using Irony for language integration into Visual Studio.
April 20, 2009 - Recordings of presentations at 2009 Lang.NET symposium are available online. I've made a couple of presentations there, the first one about Irony. Watch it, come back with comments. Watch other recordings - very interesting stuff!

Updated Wiki: Demo Running Instructions

rivantsov — Thu, 11 Aug 2011 07:12:16 GMT

Demo instructions

If you are interested in Silverlight version of Irony, make sure you install Silverlight tools before you open Irony solution - see instructions below.
Open Irony_All.sln solution file in Visual Studio.
If you are not interested in Silverlight development, remove the project "100.IronySilverlight" from the solution.
Right-click on the project "030.Irony.GrammarExplorer" and select "Set as StartUp project" from the context menu.
Click Run button on toolbar (F5). Grammar Explorer Window opens.
Important: if you are launching the Grammar Explorer for the first time after downloading Irony and if you see that Grammars combobox is not empty, then clear it: click on the button next to the combobox and select "Remove all"
If the Grammar combobox on top is empty, click on the button next to it (or right-click the combobox) and select "Add grammar" command. In the file-open window that appears, navigate to (root)\Irony.Samples\bin\debug folder and select Irony.Samples.dll. Application will popup a small window with a list of grammars in selected assembly. Leave all lines checked and click "OK". The newly added grammars will appear in the grammar combobox.
Select a grammar (language) in top combo-box. If you see exception window with error popping up, then go back to previous step and remove all old grammars by selecting "Remove all" command.
Browse form tabs to see grammar data.
To parse source code sample, switch to "Test" tab. Click "Load..." button on top of the form. Open file dialog opens.
Navigate to <root>\Irony.Samples\SourceSamples folder. Select source file appropriate for currently selected grammar.
Source file contents are loaded into text area in the form. Click Parse button on top of the form.
The Output Syntax Tree control on the right (in Results tab in the right tab control) would contain a parsed syntax tree.
If button "Run" is enabled, click it to execute the code. The results are shown in the Output window at the bottom of the form. For Expression Evaluator grammar, the output is the result of the last expression or assignment. Interpreter for Scheme can execute more complex programs found in SourceSamples\Scheme subfolder.
Alternatively you can paste or type your own sample program into source text area.
Repeat for other selections in the Grammar combobox.

Note that Grammar Explorer restores your last language selection and source sample after you close/restart the form.

Installing Silverlight tools

If you are going to use Team Foundation Server Client, install it from here:

http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=0ed12659-3d41-4420-bbb0-a46e51bfca86

Install Visual Studio 2010
Install Silverlight 4 Tools (from http://www.microsoft.com/download/en/details.aspx?id=18149)

Updated Wiki: Demo Running Instructions

rivantsov — Thu, 11 Aug 2011 07:11:53 GMT

Demo instructions

If you are interested in Silverlight version of Irony, make sure you install Silverlight tools before you open Irony solution - see instructions below.
Open Irony_All.sln solution file in Visual Studio.
If you are not interested in Silverlight development, remove the project "100.IronySilverlight" from the solution.
Right-click on the project "030.Irony.GrammarExplorer" and select "Set as StartUp project" from the context menu.
Click Run button on toolbar (F5). Grammar Explorer Window opens.
Important: if you are launching the Grammar Explorer for the first time after downloading Irony and if you see that Grammars combobox is not empty, then clear it: click on the button next to the combobox and select "Remove all"
If the Grammar combobox on top is empty, click on the button next to it (or right-click the combobox) and select "Add grammar" command. In the file-open window that appears, navigate to (root)\Irony.Samples\bin\debug folder and select Irony.Samples.dll. Application will popup a small window with a list of grammars in selected assembly. Leave all lines checked and click "OK". The newly added grammars will appear in the grammar combobox.
Select a grammar (language) in top combo-box. If you see exception window with error popping up, then go back to previous step and remove all old grammars by selecting "Remove all" command.
Browse form tabs to see grammar data.
To parse source code sample, switch to "Test" tab. Click "Load..." button on top of the form. Open file dialog opens.
Navigate to <root>\Irony.Samples\SourceSamples folder. Select source file appropriate for currently selected grammar.
Source file contents are loaded into text area in the form. Click Parse button on top of the form.
The Output Syntax Tree control on the right (in Results tab in the right tab control) would contain a parsed syntax tree.
If button "Run" is enabled, click it to execute the code. The results are shown in the Output window at the bottom of the form. For Expression Evaluator grammar, the output is the result of the last expression or assignment. Interpreter for Scheme can execute more complex programs found in SourceSamples\Scheme subfolder.
Alternatively you can paste or type your own sample program into source text area.
Repeat for other selections in the Grammar combobox.

Note that Grammar Explorer restores your last language selection and source sample after you close/restart the form.

Installing Silverlight tools

If you are going to use Team Foundation Server Client, install it from here:

http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=0ed12659-3d41-4420-bbb0-a46e51bfca86

Install Visual Studio 2010
Install Silverlight 4 Tools (fromhttp://www.microsoft.com/download/en/details.aspx?id=18149)

Updated Wiki: Home

rivantsov — Thu, 04 Aug 2011 18:24:04 GMT

Writing your first Domain-Specific Language - two articles by Daniel Flower at CodeProject.com. An excellent introduction to using Irony for parsing "little" application-specific languages - DSLs.
A Google-like Full Text Search - an article by Michael Coles about Google-to-SQL Server full-text search query converter based on Irony.
02/10/2009 - new article on CodeProject: Writing your first Visual Studio Language Service by Ben Morrison about using Irony for language integration into Visual Studio.
April 20, 2009 - Recordings of presentations at 2009 Lang.NET symposium are available online. I've made a couple of presentations there, the first one about Irony. Watch it, come back with comments. Watch other recordings - very interesting stuff!
May 2, 2009. Started a blog about Irony: http://irony-roman.blogspot.com/. The first post is about changes in April code update. Enjoy it!
Oct 12, 2009. Updated the release download version
June 6, 2010. A wikibook for documenting Irony : Irony - Language Implementation Kit (http://en.wikibooks.org/wiki/Irony-LanguageImplementationKit). Everybody is welcome to contribute!

More information
Contributors
Willing to Contribute?
Demo Running Instructions
Expression Grammar sample
Irony presentation at LangNET 2009 symposium
Irony blog
Wikibook: Irony - Language Implementation Kit
My other project: VITA - ORM done right

Updated Wiki: Contributors

rivantsov — Tue, 02 Aug 2011 21:38:01 GMT

Contributors

The following people made contributions to Irony project, and I would like to express my gratitude to them:

Alexey Yakovlev (yallie)

Contributed Refal programming language implementation.
Implemented auto-reloading on compile of grammar assemblies in Grammar Explorer.
Implemented LineContinuationTerminal.
Implemented semi-automatic parser conflict resolution from token preview (08/02/2011 - currently in review).

William Horner (wmh): contributed implementation of auto-compiled delegates for AST node creation.
Philipp Serr: completed NumberLiteral implementation with advanced features for c#, python, VB
CodePlex user notmasteryet: improved GwBasic grammar to make it match the syntax rules of GwBasic
Benjamen Morrison: created an implementation of Language Services for Visual Studio integration based on Irony. This code is not a part of Irony codebase (yet), but you can find it in the article in CodeProject.com
Michael Coles - created SearchGrammar for converting Google-like search queries into MS SQL Server full-text search syntax. The code is used in his book "Pro Full-Text Search in SQL Server 2008".
Kirill Osenkov: created IronySilverlight, Irony version for Silverlight environment

Updated Wiki: Contributors

rivantsov — Tue, 02 Aug 2011 21:34:14 GMT

Contributors

The following people made contributions to Irony project, and I would like to express my gratitude to them:

Alexey Yakovlev (yallie)

Contributed Refal programming language implementation.
Implemented auto-reloading on compile of grammar assemblies in Grammar Explorer.
Implemented LineContinuationTerminal.
Implemented semi-automatic parser conflict resolution from token preview (08/02/2011 - currently in review).

user wmh: contributed implementation of auto-compiled delegates for AST node creation.
Philipp Serr: completed NumberLiteral implementation with advanced features for c#, python, VB
CodePlex user notmasteryet: improved GwBasic grammar to make it match the syntax rules of GwBasic
Benjamen Morrison: created an implementation of Language Services for Visual Studio integration based on Irony. This code is not a part of Irony codebase (yet), but you can find it in the article in CodeProject.com
Michael Coles - created SearchGrammar for converting Google-like search queries into MS SQL Server full-text search syntax. The code is used in his book "Pro Full-Text Search in SQL Server 2008".
Kirill Osenkov: created IronySilverlight, Irony version for Silverlight environment

Updated Wiki: Contributors

rivantsov — Tue, 02 Aug 2011 21:32:39 GMT

Contributors

The following people made contributions to Irony project, and I would like to express my gratitude to them:

Alexey Yakovlev (yallie)

Contributed Refal programming language implementation.
Implemented auto-reloading on compile of grammar assemblies in Grammar Explorer.
Implemented LineContinuationTerminal.
Implemented semi-automatic parser conflict resolution from token preview (08/02/2011 - currently in review).

Philipp Serr: completed NumberLiteral implementation with advanced features for c#, python, VB
CodePlex user notmasteryet: improved GwBasic grammar to make it match the syntax rules of GwBasic
Benjamen Morrison: created an implementation of Language Services for Visual Studio integration based on Irony. This code is not a part of Irony codebase (yet), but you can find it in the article in CodeProject.com
Michael Coles - created SearchGrammar for converting Google-like search queries into MS SQL Server full-text search syntax. The code is used in his book "Pro Full-Text Search in SQL Server 2008".
Kirill Osenkov: created IronySilverlight, Irony version for Silverlight environment

Updated Wiki: ContribProjects

rivantsov — Tue, 02 Aug 2011 21:28:36 GMT

Willing to Contribute? Here is a To-Do list for Release

July 07, 2011.
Hi everybody!
These are things that, in my opinion, need to be done before we declare the Release. Any help and contributions are welcome!
For a while, I will NOT be working on any of these items, or in this area - so feel free to pick any item and start hacking. I will be finishing Interpreter.
If you see something you would like to work on, make sure you are in contact with other people who may be working on the same item. Once you selected the project, find an existing discussion related to it. Either follow the link to discussion thread in the project item (if it is enabled), or search Discussions page for a thread about this item. If there's no existing thread, then start a new one - set the title to match the item title. Post a message to this discussion thread announcing your intent to work on some item, with a link to item's discussion.
I will update the link to discussion thread on this page.

Scanning and Terminals

~~P1: Create LineContinuationTerminal~~ - DONE, http://irony.codeplex.com/discussions/265782
A terminal class to handle continuation symbols like "_" in VB, use it in GWBasic.
Note: MiniPython grammar in Samples uses CodeOutlineFilter that handles continuation symbol "\". But for simpler case like GwBasic it seems overcomplicated, better have a simple terminal just for this.

P2: Handling embedded documentation (Xml comments) (Discussion thread)
A terminal to handle Xml comments or other types of embedded documentation ("=begin/=end" in Ruby, DocLines in Python). The terminal (TokenFilter?) should assemble documentation block from multiple lines and put it into ParseTreeNode.Documentation field (to be added) of the subsequent parse tree node.

P3: Implement IncludeText support (Discussion thread)
Facility to handle "include" statement, when you include text directly into the source (not reference external module). See discussion http://irony.codeplex.com/Thread/View.aspx?ThreadId=204640

P4: Fancy string types (Discussion thread)
Either extend StringLiteral, or create a new terminal to handle the following:

Lua: [[ ... ]] string type; note: fist newline is ignored. Also the start end pair might be with "=" added: [===[ .... ]===] ; same for comment --[[ ... ]]
Ruby: '%' notation (http://en.wikibooks.org/wiki/Ruby_Programming/Syntax/Literals)

       myString = %&This is my String&
	%Q/This is the same as a double-quoted string./    # '%Q' is treated as double-quoted string

Interesting case is '%w': %w(foo bar baz) is equivalent to ['foo', 'bar', 'baz'] - maybe a separate terminal for handling this?

P5: Handling implied "+" between string literals. (Discussion thread)
Ruby allows missing "+" operator in concatenating strings:

		myString = "Welcome " "to " "Ruby!"
		=> "Welcome to Ruby!"

Add support for this to StringLiteral - new option flag; if it is set, the matching code tries to search beyond the ending quote and checks if it is followed by other literal; if yes - the string is joined in one token.

P6: Implement HereDoc terminal (Ruby) (Discussion thread)
Implement a terminal class to handle embedded text like HereDoc in Ruby (similar things in other languages?).
Ruby HereDoc element is treated as double-quoted string; '<<-' allows indenting ending tag; '<<' does not;
Note that Ruby allows starting multiple HereDocs on a single line

		puts <<BEGIN + "<--- middle --->\n" + <<END
		This is the beginning:
		BEGIN
		And now it is over!
		END
		
		# this equals this expression:
		puts "This is the beginning:\n<--- middle --->\nAnd now it is over!"

P7: Create reduced Ruby grammar (Discussion thread)
To test all these fancy terminals mentioned in previous sections (Ruby variations) - would be nice to have some reduced version of Ruby grammar - call it MiniRuby?

P8: Add RegEx validation pattern to Identifier terminal; support for specific values (Discussion thread)
Some languages have certain rules about identifiers; for ex, 'Oneletter+OneDigit' in GwBasic. Add a property to IdentifierTerminal, Regex expression, to validate the identifier.
On a similar theme: Scheme has a notion of "Peculiar identifier" in its syntax definition - identifier with a fixed list of values:
Peculiar identifier = +|-|...
See how this can be supported. Maybe Regex pattern listing all variations?

P9: Support for nested comments (Discussion thread)
Examples: {- and -} in Haskell, similar thing in Scheme with "'(...)" - commenting out blocks of code with nested parenthesis.
Add support for this to CommentTerminal.

P10: Support for "special" treatment of the first line in code file (Discussion thread)
Lua- first line in file ignored if it starts with #; similar in CSV files - the first line has field names
One way to implement: add a LanguageFlag, and if it is set, Scanner calls some special overridable method in Grammar: HandleFirstLine(....)

P11: Implement BackgroundTextTerminal (Discussion thread)
A special terminal for grabbing everything that does not fall into other terminals. Will be used in Wiki and template parsers.
Let's look at Wiki terminals as an example. It has a bunch of terminals with certain start symbols, like "*" for bold, etc.
Scanner select a terminal based on current input symbol, trying to match it to one of First symbols of registered terminals. If it fails to match to any, it should call Background terminal (this can be done probably by making it a "fallback" terminal - the one that does not declare any test symbols). Now the interesting stuff. The background terminal should "eat" the text until it hits any of the "First" symbols of other terminals. At this point it stops, produces token and yields control to Scanner - which will try to select a terminal again.
To be able to do this (stop at other's firsts), the background terminal must be able to "spy" on other terminals and grab all their "first" symbols at initialization.

P12: Terminal for handling inlined Xml (Discussion thread)
A terminal to scan inlined Xml (Scala).

Miscellaneous projects (more challenging and requiring more effort)

~~P21: Implement fast compiled delegate to create AST nodes~~ DONE: http://irony.codeplex.com/discussions/265806
AstNodes are created by Parser using reflection, which is slow. Implement creating dynamic compiled delegates calling AST node constructors. For an example, see Vita project, EntityClassBuilder class, method InstanceCreatorMethod - creates compiled method to create entity instances. Plan: NonTerminal should have a property Creator - pointer to delegate creating the Ast instance. If it is null (default), then the method is built by ParserDataBuilder. Then this method is used by Parser for creating node instances. Place all static creator methods in one static class generated at runtime.
Note that there is already a method pointer for custom AST node creation. This new one is an addition, for the case when grammar provides AstNode type only.

P22: Switch to .NET big int, complex, rational in VS2010 (Discussion thread)
BigInteger, Complex are supported in .NET 4.0. Implementation of ration numbers (BigRational) is available here: http://bcl.codeplex.com.
A good place to test is Scheme grammar - Scheme supports all these data types.

P23: Console window for Grammar Explorer (Discussion thread)
Create and integrate Console control in Grammar Explorer, to show interactive console directly inside in the form, instead of read-only Runtime Output window.

P24: Pretty print facility (Discussion thread)
Implement "pretty printer" - some way to output parse tree in a nice formatted form. The formatting options should be provided through some setup class (like indentation, etc)

P25: Specifying custom infix operator precedence and associativity on per-file basis. (Discussion thread)
Haskell allows this: infix, infixl, infixr methods in Haskell.

P26: Finish LINQ syntax implementation in c# grammar. (Discussion thread)
This is mostly to explore additional twist - scoped reserved words.
Extract from c# language spec, section 7.16.1:

Query expressions contain a number of “contextual keywords”, i.e., identifiers that have special meaning in a given context. Specifically these are from, where, join, on, equals, into, let, orderby, ascending, descending, select, group and by. In order to avoid ambiguities in query expressions caused by mixed use of these identifiers as keywords or simple names, these identifiers are considered keywords when occurring anywhere within a query expression.
For this purpose, a query expression is any expression that starts with “from identifier” followed by any token except “;”, “=” or “,”.

This introduces a concept of "scoped" reserved words: some keywords are reserved words (in Irony's meaning) only inside some scope/statement. Suggestions on implementation?

P27: Refactor/rebuild unit tests projects. (Discussion thread)
This is a big one. The current state is a shameful mess. Existing unit tests make quick shortcuts to instantiate terminals and call TryParse directly.
We need to implement more complete tests, with test grammars, systematically covering all aspects of functionality.

Advanced Research Projects

P31: Survey popular languages
Survey modern languages and identify which facilities are missing in Irony to build parsers for these languages (mainly terminals). Suggested languages to look at: Scheme, Python, Ruby, GwBasic, Lua, Tcl, JavaScript, Clojure, Scala, Boo,Haskell, Erlang, Groovy, Smalltalk, PHP, Go (new Google language).
All these are modern and popular languages - but why not include older ones - like Cobol? Even if we never write Cobol compiler, somebody may need to write colorizer for Cobol editor.

~~P32: Token Preview -based Semi-Automatic Conflict Resolution.~~ (DONE)
This topic deserves its own page: Automatic Conflict Resolution.

P33: Macro facilities
See how to implement macro facilities, starting from plain c macros up to advanced stuff like Scheme Macro system.

P34: Template processor
Parser for template files, which include "text" and embedded script commands (Ruby rhtml format as an example).
Things to consider - embedded grammars or sub-grammars, a separate grammar attached to some terminal that will be used to parse sub-sections of the text.

P35: Basic code analysis algorithms
Implement some basic code analysis - "Lattices" and all that stuff. Maybe not for this release, but more long-term. The goal is to implement algorithms for detecting loops, uninitialized variables, unused code, etc. The most approachable introduction to the subject I found in "Advanced Compiler Design Implementation" by Steven Muchnick.

Updated Wiki: ContribProjects

rivantsov — Tue, 02 Aug 2011 21:25:39 GMT

Willing to Contribute? Here is a To-Do list for Release

Scanning and Terminals

P1: Create LineContinuationTerminal (Discussion thread)
A terminal class to handle continuation symbols like "_" in VB, use it in GWBasic.
Note: MiniPython grammar in Samples uses CodeOutlineFilter that handles continuation symbol "\". But for simpler case like GwBasic it seems overcomplicated, better have a simple terminal just for this.

P2: Handling embedded documentation (Xml comments) (Discussion thread)
A terminal to handle Xml comments or other types of embedded documentation ("=begin/=end" in Ruby, DocLines in Python). The terminal (TokenFilter?) should assemble documentation block from multiple lines and put it into ParseTreeNode.Documentation field (to be added) of the subsequent parse tree node.

P3: Implement IncludeText support (Discussion thread)
Facility to handle "include" statement, when you include text directly into the source (not reference external module). See discussion http://irony.codeplex.com/Thread/View.aspx?ThreadId=204640

P4: Fancy string types (Discussion thread)
Either extend StringLiteral, or create a new terminal to handle the following:

Lua: [[ ... ]] string type; note: fist newline is ignored. Also the start end pair might be with "=" added: [===[ .... ]===] ; same for comment --[[ ... ]]
Ruby: '%' notation (http://en.wikibooks.org/wiki/Ruby_Programming/Syntax/Literals)

       myString = %&This is my String&
	%Q/This is the same as a double-quoted string./    # '%Q' is treated as double-quoted string

		myString = "Welcome " "to " "Ruby!"
		=> "Welcome to Ruby!"

		puts <<BEGIN + "<--- middle --->\n" + <<END
		This is the beginning:
		BEGIN
		And now it is over!
		END
		
		# this equals this expression:
		puts "This is the beginning:\n<--- middle --->\nAnd now it is over!"

Miscellaneous projects (more challenging and requiring more effort)

P21: Implement fast compiled delegate to create AST nodes (Discussion thread)
AstNodes are created by Parser using reflection, which is slow. Implement creating dynamic compiled delegates calling AST node constructors. For an example, see Vita project, EntityClassBuilder class, method InstanceCreatorMethod - creates compiled method to create entity instances. Plan: NonTerminal should have a property Creator - pointer to delegate creating the Ast instance. If it is null (default), then the method is built by ParserDataBuilder. Then this method is used by Parser for creating node instances. Place all static creator methods in one static class generated at runtime.
Note that there is already a method pointer for custom AST node creation. This new one is an addition, for the case when grammar provides AstNode type only.

P22: Switch to .NET big int, complex, rational in VS2010 (Discussion thread)
BigInteger, Complex are supported in .NET 4.0. Implementation of ration numbers (BigRational) is available here: http://bcl.codeplex.com.
A good place to test is Scheme grammar - Scheme supports all these data types.

P23: Console window for Grammar Explorer (Discussion thread)
Create and integrate Console control in Grammar Explorer, to show interactive console directly inside in the form, instead of read-only Runtime Output window.

P24: Pretty print facility (Discussion thread)
Implement "pretty printer" - some way to output parse tree in a nice formatted form. The formatting options should be provided through some setup class (like indentation, etc)

P25: Specifying custom infix operator precedence and associativity on per-file basis. (Discussion thread)
Haskell allows this: infix, infixl, infixr methods in Haskell.

P26: Finish LINQ syntax implementation in c# grammar. (Discussion thread)
This is mostly to explore additional twist - scoped reserved words.
Extract from c# language spec, section 7.16.1:

Query expressions contain a number of “contextual keywords”, i.e., identifiers that have special meaning in a given context. Specifically these are from, where, join, on, equals, into, let, orderby, ascending, descending, select, group and by. In order to avoid ambiguities in query expressions caused by mixed use of these identifiers as keywords or simple names, these identifiers are considered keywords when occurring anywhere within a query expression.
For this purpose, a query expression is any expression that starts with “from identifier” followed by any token except “;”, “=” or “,”.

This introduces a concept of "scoped" reserved words: some keywords are reserved words (in Irony's meaning) only inside some scope/statement. Suggestions on implementation?

P27: Refactor/rebuild unit tests projects. (Discussion thread)
This is a big one. The current state is a shameful mess. Existing unit tests make quick shortcuts to instantiate terminals and call TryParse directly.
We need to implement more complete tests, with test grammars, systematically covering all aspects of functionality.

Advanced Research Projects

Updated Wiki: Contrib_ConflictResolution

rivantsov — Tue, 02 Aug 2011 21:24:07 GMT

Token Preview -based Semi-Automatic Conflict Resolution

(first version completed, see here: http://irony.codeplex.com/discussions/267217)

Suggested research/implementation project.

There is one very common type of reduce/reduce conflict that comes up in languages like c# and java.
I'll show it using a simple language as an example. The language recognizes a list of declarations, for ex:

    public readonly string SomeField;
    private int SomeProp {}; 
    public object SomeMethod(){};

Here is an Irony grammar for this language:

  public class ConflictResolutionTestGrammar : Grammar {
    public ConflictResolutionTestGrammar() : base(true) {
      var name = new IdentifierTerminal("id");

      var stmt = new NonTerminal("Statement");
      var stmtList = new NonTerminal("StatementList");
      var fieldModifier = new NonTerminal("fieldModifier");
      var propModifier = new NonTerminal("propModifier");
      var methodModifier = new NonTerminal("methodModifier");
      var fieldModifierList = new NonTerminal("fieldModifierList");
      var propModifierList = new NonTerminal("propModifierList");
      var methodModifierList = new NonTerminal("methodModifierList");
      var fieldDef = new NonTerminal("fieldDef");
      var propDef = new NonTerminal("propDef");
      var methodDef = new NonTerminal("methodDef");

      //Rules
      this.Root = stmtList;
      stmtList.Rule = MakePlusRule(stmtList, stmt);
      stmt.Rule = fieldDef | propDef | methodDef;
      fieldDef.Rule = fieldModifierList + name + name + ";";
      propDef.Rule = propModifierList + name + name + "{" + "}";
      methodDef.Rule = methodModifierList + name + name + "(" + ")" + "{" + "}";
      fieldModifierList.Rule = MakeStarRule(fieldModifierList, fieldModifier);
      propModifierList.Rule = MakeStarRule(propModifierList, propModifier);
      methodModifierList.Rule = MakeStarRule(methodModifierList, methodModifier);
      
      // That's the key of the problem: 3 modifiers have common members
      //   so parser automaton has hard time deciding which modifiers list to produce - 
      //   is it a field, prop or method we are beginning to parse?
      fieldModifier.Rule = ToTerm("public") | "private" | "readonly" | "volatile";
      propModifier.Rule = ToTerm("public") | "private" | "readonly" | "override";
      methodModifier.Rule = ToTerm("public") | "private" | "override";

      MarkReservedWords("public", "private", "readonly", "volatile", "override");   
    }

Each statement type is defined the same way: modifiers list followed by typename, name, finally ending with either ";", "{}", or "(){}". The trouble is that modifiers are different for each statement type - share some keywords, but each is different. Naturally, we define different -List and -Modifier non-terminals for modifiers. And here comes the conflict: when parser sees the first word of a statement, like "public", it cannot know which modifierList it belongs to, and which non-terminal to reduce, out of 3.
Here's grammar error in S0, the same is in S1:
Reduce-reduce conflict. State S0, lookaheads: id public private readonly override. Selected reduce on first production in conflict set.

Here are reeduce items in state S0 that are in conflict:

    fieldModifierList -> · [id public private readonly volatile]
    propModifierList -> · [id public private readonly override]
    methodModifierList -> · [id public private override]

The idea for the fix is simple: make Parser run ahead and check the first token it finds out of the set [ ; ( { ]. If it finds ";" first out of 3, it is field definition, if { then it is property, and so on.
We need a way to provide parser with these hints, and let parser builder know that we're OK, the conflict is resolved.

(important: the implementation section was edited on 07/20/2011)
Implementation suggestion: implement a grammar hint TokenPreviewHint, with two parameters:

ComesFirst - a symbol to find first;
BeforeSymbols - before these symbols; we can use either array (params array) or space-delimited list in a single string.

When this hint is used in production in Shift position (not at the end), it is "shift-if" hint; correspondingly, when the hint is at the end of production, it is "reduce-if" instruction.
Two helper methods can be added, that return this hint:

ShiftIf(comesFirst, beforeSymbols); - to use in shift position.
ReduceIf(comesFirst, beforeSymbols); - to use in reduce position.

Note that the example we just looked at (modifiers list) is in fact a special case. When we create lists using MakePlusRule or MakeStarRule, we do not form the productions explicitly, these methods do this for us. So for this case, we need some additional way to "inject" these hints. Probably, some helper method ReduceIf on non-terminal that adds this hint to all productions in a non-terminal:

     fieldModifierList.ReduceIf(";", "; { ("); 
     propModifierList.ReduceIf("{", "; { ("); 
     methodModifierList.ReduceIf("(", "; { (");

The first parameter is a symbol to find first, out of symbols/terminals listed in the second parameter (as a space delimited string). Note that we need different ways to specify terminals - as string literals like shown above, or as references to terminal variables used in grammar definitions.
Another thing - we may need to provide extra symbol(s) marking when to stop searching so that preview does not go too far - in case we are parsing source with errors. Example: user is typing text in editor, missed one symbol, while highlighter is parsing the text - we do not want parser to run too far. In our case, the termination symbols might be " or ")";
Implementing this behavior is step one - when user knows how to resolve the conflict, and he provides explicit instructions. More difficult is next step: actually verifying that with provided hints the conflict is gone for all cases. And finally, it would be nice to have automatic advice: when user runs into conflict like this the first time, Irony automatically discovers the way to resolve it using this technique, and give an advise to the user, including the symbols to use in resulution.

Updated Wiki: Home

rivantsov — Mon, 01 Aug 2011 03:22:42 GMT

Writing your first Domain-Specific Language - two articles by Daniel Flower at CodeProject.com. An excellent introduction to using Irony for parsing "little" application-specific languages - DSLs.
A Google-like Full Text Search - an article by Michael Coles about Google-to-SQL Server full-text search query converter based on Irony.
02/10/2009 - new article on CodeProject: Writing your first Visual Studio Language Service by Ben Morrison about using Irony for language integration into Visual Studio.
April 20, 2009 - Recordings of presentations at 2009 Lang.NET symposium are available online. I've made a couple of presentations there, the first one about Irony. Watch it, come back with comments. Watch other recordings - very interesting stuff!
May 2, 2009. Started a blog about Irony: http://irony-roman.blogspot.com/. The first post is about changes in April code update. Enjoy it!
Oct 12, 2009. Updated the release download version
June 6, 2010. A wikibook for documenting Irony : Irony - Language Implementation Kit (http://en.wikibooks.org/wiki/Irony-LanguageImplementationKit). Everybody is welcome to contribute!
Oct 25 2010. Irony is on hold. More details here.