Changeset 15610, Nov 5, 2008 - Syntax coloring
* Support for syntax coloring in customized editor control
* Grammar Explorer form is finally fixed and allows loading arbitrary grammars from compiled assemblies, instead of hard-coded list as it was before
Changeset 12333, Sept 1, 2008 - language runtime
* Implemented LanguageRuntime with dynamic operator dispatch - base class for custom language runtimes. One example is Scheme sample and SchemeRuntime class - a customization of Irony's LanguageRuntime class. Another example - CalcGrammar in Tutorial, uses
default runtime implementation and provides evaluation of simple arithmetic expressions.
* Big refactoring: moved LALR parsing classes to a separate folder/namespace; making it possible to create other, non-LALR parsers with the same Grammar-construction methods and Scanner implementation. Planning to implement NLALR (Non-canonical LALR) parser
algorithm. As part of refactoring, separated scanner-related data from parser data into ScannerControlData class.
* Implemented GrammarHint class, to allow adding hints for parser in grammar expressions - for example, to explicitly set an action in case of shift/reduce or reduce/reduce conflicts. Grammar.PreferShiftHere() and Grammar.ReduceThis() methods can be used to
create hints inside grammar expressions. See GwBasic sample grammar, ELSECLAUSEOPT and FUN_CALL elements for examples of hint use; also see SchemeGrammar.
* Cleaned up Scheme grammar, removed LALR conflicts by rearranging NonTerminals and adding grammar hints. Scheme now has real runtime implementation, supports various data types for arithmetic operations.
* Bug fix: double-escape char at the end of the string literal was not parsed correctly.
* Applied SecurityTransparent attribute to Irony assembly; more info: http://msdn.microsoft.com/en-us/library/bb397858.aspx
* RegexTerminal - enhanced following suggestions by user sakana280.
* CodePlex member "notmasteryet" submitted improved GWBasic grammar - thanks! I've tweaked it a bit to fix the conflicts.
Changeset 10877, May 16, 2008 - it's all about performance
* Changed GrammarData to use StringSets instead of StringLists for lookaheads computations - 2x+ performance improvement in parser initialization
* Dynamic operator dispatch engine - implementation of binary operators (+, -, etc.) and dynamic dispatch to implementation based on arg types. See
Runtime/OperatorDispatch.cs file. Not actually used by any samples yet, only in unit test class - see
OperatorDispatchTests.cs file for sample use and performance evaluation. And performance is not bad - less than 200 ns per operation on 2Hrz machine.
* Tests moved to VS Test framework - now run directly in Visual Studio, without NUnit.
* Added CreateSchemeNumber method to TerminalFactory (not quite R6RS yet though)
Changeset 10663, May 5, 2008 - a scheme for interpreter and interpeter for Scheme
The first elements of interpeter engine. Implementation for trivial subset of Scheme - advanced enough to run 99 bottles.
More details are coming.
Note: Python and Ruby grammars are temporarily excluded from grammar samples. These grammar classes use the versions of Kleene operators (Star, Plus) that had been deprecated - so until they are updated, they are not available in samples.
Changeset 10309, April 17, 2008 - It is 2008, finally
* Solution migrated to VS 2008/.NET 3.5. Tests are still through NUnit - will be moved to MSTest later
* Added GwBasic grammar
* Improved MakeStarRule implementation with null delimiter - now introduces less conflicts
* Renamed Grammar.ExtraTerminals list to Grammar.NonGrammarTerminals. Added IsNonGrammar as TermOption; for terminals in NonGrammarTerminals this flag is set automatically; and in this case the parser ignores tokens of this type. This usually happens with Comment
* Fixed CommentTerminal to allow line comments to be the last tokens in the file, without final line break.
Changeset 10146, April 12, 2008 - Time to get real
* First REAL language grammar - c# 3.0 grammar. All language features except LINQ queries and preprocessor directives.
* Philipp Serr completed implementation of NumberTerminal with support for Complex numbers, Big Integers, and some advanced features required by c#, Python, VB
* Deprecated Kleene operators methods BnfTerm.Plus, Star(). Introduced Grammar.MakePlusRule, MakeStarRule methods that should be used for creating lists.
* Removed ReservedWord identifier, introduced Token.IsKeyword flag; renamed ReservedWords to Keywords, created internal hash table for quick keyword lookup.
* Refactored GrammarDataBuilder method for performance improvements
* Added Parser.PreviewSymbols method for resolving conflicts; method finds which of the tokens in a given list is coming first
* Add LineTerminators property to Grammar, to explicitly set characters used by Scanner for line counting
* CommentTerminal now allows multiple end symbols - used in c# LineComment terminal which can end with one of several line terminator symbols defined in c#.
* Added AstNode.Span property of type SourceSpan (StartLocation + Length) to identify the span of source code for the node
* Created AstNodeArgs struct - container for AstNode constructor arguments. The purpose is to simplify custom AstNode signatures
* Created static method CompilerContext.CreateDummy for creating dummy context for unit tests
* Create Token.CreateMultitoken method, to allow terminals to return multiple tokens packed in one.
* Improved error handling in GrammarDataBuilder & Explorer
* Added Grammar.FallbackTerminals - terminals without explicitly declared Firsts (start chars) or having start chars defined by Unicode category (like c# Identifier)
* Added skeleton classes and methods for language Runtime implementation.
Changeset 9122, Mar 7, 2008 - Terminals, part II
* Added CompoundTerminalBase class as an abstract base class for terminals having 3-part structure - prefix-body-suffix. Refactored Identifier, StringLiteral and NumberLiteral classes to inherit from this class. For more explanation of the concept see comments
on top of the source file.
* Added a property Token.Details (of type ScanDetails) containing (optionally) the detailed information from scanning compound tokens: prefix, suffix, flags, etc.
* Full-featured implementation of IdentifierTerminal supporting all fancy features of c# 3.0 identifiers - prefixes (@), escaped chars, non-ASCII charsets, specifying allowed chars by unicode categories (including formatting chars)
* Removed NodeCreator delegate, added NodeCreating, NodeCreated events on NonTerminal - following standard template
* Renamed BnfElement to BnfTerm, BnfFlags to TermOptions
* Added DebuggerStepThrough attribute to properties and one-liners
* Added ISourceStream.MatchSymbol method - simplifies code in terminals
* Added Search box to GrammarExplorer for searching text in top/active textbox - contributed by Andrew Bradnan.
Changeset 8968, Mar 3, 2008 - It's all about Terminals
* Fully implemented StringLiteral, now supports escaped characters, multi-line strings; doubled quote inside as single.
* Fully implemented NumberTerminal - exponent support; custom exponent symbols; support for hex prefixes (0x...), type suffixes; support for BIG INTEGERS
* New TerminalFactory static class with methods for creating Number and String literals for c#, VB, Python
* Unit tests for terminals - many of them. See Readme file for instructions on running unit tests.
* Comment terminal - removed a special treatment of Line comments; comment now is StartSymbol...EndSymbol;
line comments fit well into this, for example in c#: StartSymbol="//", EndSymbol = "\n"
* Removed CharLiteral class - for c# char literal use StringLiteral with IsChar option
* Added Script.NET grammar to the list of sample grammars
* Removed ErrorTerminal class, using base Terminal class for Grammar.Error singleton
Changeset 7937, Feb 15
* Repaired terminal sorting for scanning
* Fixed terminals sorting for display
* Fixed BnfElement.Star(delimiter) method - caused false grammar conflicts
* Fixed parsing for case-insensitive grammars - now should work correctly
* Fixed exception when NonTerminal's Rule property was not assigned - now reports error nicely and cancels the process
* Refactored Scanner to accept ISourceStream instead of SourceFile instance; a move towards using general Stream as source
* Refactored handling punctuation symbols - added IsPunctuation flag to BnfElement and removed Grammar.PunctuationSymbols symbols collection; use Grammar.RegisterPunctuation method for setting the flag in punctuation symbols and even NonTerminals
* Refactored operator symbols handling - added IsOperator flag, precedence and associativity properties directly to SymbolTerminal
* Streamlined AstNode creation process - moved most of the logic to Parser.CreateNode method, cleaned it up. (Python grammar is a bit broken as a result - AST tree shows some "extra" symbols)
* Deprecated GenericNode - AstNode is main base class for all Ast nodes
* Added facilities for VS integration: Scanner.GetNext method, returning tokens one-by-one; BraceMatchTokenFilter builds full list of brace pairs.
* Basic AST Visitor facility: AstNode.AcceptVisitor(IAstVisitor)
* Refactored internal representation of BnfExpression implementation, now without ExpressionType enumeration property
* Improved grammar data presentation by removing \b symbol from names
* Renamed NotTerminal.Expression property to Rule
* Added Values dictionary to CompilerContext
* Added Compiler property to CompilerContext
* Added Attributes property (dictionary) to AstNode class, to store any custom values associated with node/token
* Implemented basic error reporting and recovery
* Scanner error recovery: added "Delimiters" list to Grammar: on error scanner skips all until whitespace or delimiter.
* Added ErrorRule property to Non-terminal to define error recovery paths. Error expressions can also be mixed with normal expressions in Rule property
* Added ErrorTerminal and SyntaxError singleton in Grammar, to be used in error rules
* Added TryMatch method to grammar as backup for token recognition; you can use this method to create "fancy" tokens - those that use extended character sets for example and don't have limited set of prefixes.
* Added error rules to Ruby grammar and source sample with errors for Ruby
* Added Terminal parameter to MatchHandler delegate to provide reference to caller object