irony Forum Rss Feed

New Post: freeTextLiteral

kappos — Sat, 21 Nov 2009 11:29:33 GMT

Hello,

i need some help.

i would parse this structur:

class test
{
Content
}

Content is free text.

I do it with this definition:

var compilation_unit = new NonTerminal("compilation_unit");
var class_declaration = new NonTerminal("class_declaration");
var class_body = new NonTerminal("class_body");
var freeTextLiteral = new FreeTextLiteral("freeTextLiteral", "\n", "\r", Environment.NewLine);
var description = new NonTerminal("description");
IdentifierTerminal identifier = TerminalFactory.CreateCSharpIdentifier("Identifier");

KeyTerm Lbr = ToTerm("{");
KeyTerm Rbr = ToTerm("}");

this.Root = compilation_unit;
compilation_unit.Rule=  class_declaration;
class_declaration.Rule = "class" + identifier + Lbr +  class_body + Rbr;
class_body.Rule = description;
description.Rule = Empty | freeTextLiteral;

But it doesnt work. 
The error is: "Syntax error, expected: freeTextLiteral. }"

The Testdata is:
class test
{
dsfsdfsdf
}

Can someone help me?

New Post: Need help with delimited values

rivantsov — Fri, 20 Nov 2009 20:04:22 GMT

I will publish a sample grammar using DsvTerminal for reading some comma-delimited files, it will be clear from an example I hope how to use it.

good luck

New Post: Need help with delimited values

mindcore — Fri, 20 Nov 2009 19:30:21 GMT

Great, that should be helpful and reduce my work. I will continue on my original path and use the ParsingContext.Values dictionary to pass custom data between calls.

On a related note, what is an example on where the DsvLiteral would but used? I know it wouldn't help in my scenario, but while researching, I couldn't think of any practical scenario where I would have some delimited text that's being parsed and convert all of the values to one specific System.TypeCode.

Thanks,

MindCore

New Post: Need help with delimited values

rivantsov — Fri, 20 Nov 2009 18:15:03 GMT

If you need to pass some "context" values to formula parser, you can use ParsingContext.Values dictionary - it is for storing custom context, values, whatever, and to exchange this information between terminals or between caller program and terminals or custom processing code in grammar. You can reach the ParsingContext object through parser.Context property right after you create Parser instance but before you call parser.Parse(...)

New Post: newbie question

rivantsov — Fri, 20 Nov 2009 18:09:06 GMT

Well, you should first be aware of two-step parsing process:

1. Scanning - recognizing chunks of chars like numbers, identifiers, keywords and combining them into tokens. This is also called lexical analysis, and rules for this operation are lexical rules.

2. Parsing itself - reading token stream from scanner and recognizing language constructs like expressions, statements, etc. This is syntax analysis.

When defining a grammar for a language, language spec often describes both lexical rules and syntax rules in the same manner. However, for language impelementor it is important to separate these, and identify which are lexical rules and what are syntax rules (rules over tokens). It is true for any compiler tool, and important for Irony because lexical rules are supported internally in different way. What you listed are lexical rules, rules over chars for combining them into tokens. These rules are implemented in Irony through predefined Terminal classes. So what you need to do is identify appropriate terminal classes (IdentifierTerminal, NumberLiteral, etc), instantiate them as local variables and set various options on them that match your lexical rules.

I see that you probably would need IdentifierTerminal, NumberLiteral, maybe some others.

Try to understand what are restrictions on terminals expressed by your lexical rules and set the options of Irony terminals appropriately. Follow the examples in Irony.Samples grammar, and also in TerminalFactory methods

New Post: Need help with delimited values

mindcore — Fri, 20 Nov 2009 17:16:27 GMT

I have a file that I would like to parse that is made up of pipe delimited lines of text where the first value describes the type of line it is. So for example, the file may look something like this:

Note that DA and DB are children under D1 and some fields like Description can span multiple lines, but each line type has a set number of fields.

Anyhow, I am currently using the System.String Split method and I'm leaning toward switching to Irony so that the result would be a tree and the lines with formulas would continued to be parsed and maintain awareness of their child/parent relationship. Originally, I felt that I could continue to use this design and simply parse the formulas, however, Irony seems to have no way to pass the parser meta data to maintain the needed awareness.

The only meta data that seems to be passed in is a filename string which defaults to <source> - referring to the Parse methods in Parser.cs.

So, what's the most efficient way to handle this scenario? Should I continue to use the Split method from System.String and only use the Parser for the formulas, or is there an easier way for Irony to parse the pipe delimited data and the formulas together while maintaining it's parent/child relationship?

Thanks,
MindCore

New Post: newbie question

ak756 — Fri, 20 Nov 2009 07:32:58 GMT

First sorry for the simple question. I am new to Irony and i want to build a parser for ASL(ACPI Source Language).

Here is a snippet of ASL name and pathname terms definition.

LeadNameChar := ‘A’-‘Z’ | ‘a’-‘z’ | ‘_’
DigitChar := ‘0’-‘9’
NameChar := DigitChar | LeadNameChar
RootChar := ‘\’
ParentPrefixChar := ‘^’
PathSeparatorChar := ‘.’
CommaChar := ‘,’
SemicolonDelimiter := Nothing | ‘;’

NameSeg := <LeadNameChar>
     | <LeadNameChar NameChar>
     | <LeadNameChar NameChar NameChar>
     | <LeadNameChar NameChar NameChar NameChar>

NameString := <RootChar NamePath> | <ParentPrefixChar PrefixPath NamePath> | NonEmptyNamePath
NamePath := Nothing | <NameSeg NamePathTail>
NamePathTail := Nothing | <PathSeparatorChar NameSeg NamePathTail>
NonEmptyNamePath := NameSeg | <NameSeg NamePathTail>
PrefixPath := Nothing | <ParentPrefixChar PrefixPath>

I defines the rootchar,commachar and etc as:
KeyTerm commaChar = ToTerm(",", "commaChar");
KeyTerm parentPrefixChar = ToTerm("^", "parentPrefixChar");
KeyTerm pathSeperateorChar = ToTerm(".", "pathSeperateorChar");
KeyTerm rootChar = ToTerm("\\","rootChar");

Am i right? And i don't know how to define LeadNameChar and DigitChar and so as NameSeg, Will anybody help me?

New Post: Ambiguous FreeTextLiteral

DenisKolodin — Thu, 19 Nov 2009 03:51:45 GMT

I've seen this behavior in debugger, but I've been assured that works fine ) I thought the problem is in my usage of Irony.

Undoubtedly I'll wait for the newest release where it fixed, because it's simplest then doing the same on any *cc toolkit!

Thank you! )

New Post: Ambiguous FreeTextLiteral

rivantsov — Wed, 18 Nov 2009 21:44:11 GMT

By the way, quite similar issue is reported in latest bug opened in Issue Tracker page. Need to fix it finally

Roman

New Post: Ambiguous FreeTextLiteral

rivantsov — Wed, 18 Nov 2009 21:41:48 GMT

I see the problem now. If you look at your first screenshot, in the right-bottom corner, you see token list. The second token produced by scanner is "FIRM" - an identifier token you defined for firms. It is Irony's inconsistency in dealing with terminals without prefixes, long outstanding item. The trouble is Scanner.SelectTerminals method. No-prefix terminals are picked up for consideration ONLY if there are no other candidates with prefixes. In your case there is one candidate - "Firm" identifier, so it leaves out the FREE_TEXT which is in FallbackTerminals. This is wrong, it should be fixed, I simply did not come up yet with consistent and reliable fix for this - this is actually more than a fix, it is sort of refactoring and change of entire scanning algorithm. Until now this bug manifested itself in little inconsistencies but your case is blown up by this. Sorry, will get to this ASAP. For now, add all Latin upper and lower letters to Firsts list of FreeTextLiterals you use - yes, it looks silly, and it would work only for English, but would work until the fix is there.

Sorry again

Roman

New Post: Ambiguous FreeTextLiteral

DenisKolodin — Wed, 18 Nov 2009 20:15:14 GMT

I've tried different rules with semicolon and linebreak. They didn't give the result I need. But i still one change:

          var FREE_TEXT = new FreeTextLiteral("FREE_TEXT", ";"); // "\n" removed

I made 2 screenshots how it works with grammar:

Screenshot when parsing fails

Screenshot when it works well

As you can see I add prefix for token. The prefix is '@'. And parsing works right. When free text starts with any char which can be start of terminal FIRM it doesn't work (
If i add a char to Firsts list of FreeTextLiteral it will work but when text starts with this char. But as I said I want free text can starts with any character (including latin).
It isn't good idea to add all latin chars to Firsts list, is it?
And more about parsing error. I try to parse text:

PORTFOLIO_EX The very long name of portfolio;
DESCRIPTION Any description. You can write much here;
FIRMS_LIST ALL_FIRMS;

Error happens at L,C (0, 13). It highlighted red in the parsing text.
Error message: Syntax error, expected: FREE_TEXT
Parser state: S4

All parser states is below (sorry for the size):

<small> State S0
Shift items:
PORTFOLIO' -> ·PORTFOLIO EOF
PORTFOLIO -> ·METADATA
METADATA -> ·PORTFOLIO_EX DESCRIPTION FIRMS
PORTFOLIO_EX -> ·PORTFOLIO_EX FREE_TEXT LINE_END
PORTFOLIO_EX -> ·PORTFOLIO FREE_TEXT LINE_END
Transitions: PORTFOLIO->S1, METADATA->S2, PORTFOLIO_EX->S3, PORTFOLIO_EX->S4, PORTFOLIO->S5,

State S1
Shift items:
PORTFOLIO' -> PORTFOLIO ·EOF

State S2
Reduce items:
PORTFOLIO -> METADATA · [EOF]

State S3
Shift items:
METADATA -> PORTFOLIO_EX ·DESCRIPTION FIRMS
DESCRIPTION -> ·DESCRIPTION FREE_TEXT LINE_END
Transitions: DESCRIPTION->S7, DESCRIPTION->S8,

State S4
Shift items:
PORTFOLIO_EX -> PORTFOLIO_EX ·FREE_TEXT LINE_END
Transitions: FREE_TEXT->S9,

State S5
Shift items:
PORTFOLIO_EX -> PORTFOLIO ·FREE_TEXT LINE_END
Transitions: FREE_TEXT->S10,

State S6
Reduce items:
PORTFOLIO' -> PORTFOLIO EOF · []

State S7
Shift items:
METADATA -> PORTFOLIO_EX DESCRIPTION ·FIRMS
FIRMS -> ·FIRMS_LIST FIRMS_VARIATIONS LINE_END
Transitions: FIRMS->S11, FIRMS_LIST->S12,

State S8
Shift items:
DESCRIPTION -> DESCRIPTION ·FREE_TEXT LINE_END
Transitions: FREE_TEXT->S13,

State S9
Shift items:
PORTFOLIO_EX -> PORTFOLIO_EX FREE_TEXT ·LINE_END
LINE_END -> ·SEMICOLON LF
Transitions: LINE_END->S14, SEMICOLON->S15,

State S10
Shift items:
PORTFOLIO_EX -> PORTFOLIO FREE_TEXT ·LINE_END
LINE_END -> ·SEMICOLON LF
Transitions: LINE_END->S16, SEMICOLON->S15,

State S11
Reduce items:
METADATA -> PORTFOLIO_EX DESCRIPTION FIRMS · [EOF]

State S12
Shift items:
FIRMS -> FIRMS_LIST ·FIRMS_VARIATIONS LINE_END
FIRMS_VARIATIONS -> ·ALL_FIRMS
FIRMS_VARIATIONS -> ·FIRMS_LIST
FIRMS_LIST -> ·FIRM
FIRMS_LIST -> ·FIRMS_LIST COMMA FIRM
Transitions: FIRMS_VARIATIONS->S17, ALL_FIRMS->S18, FIRMS_LIST->S19, FIRM->S20,

State S13
Shift items:
DESCRIPTION -> DESCRIPTION FREE_TEXT ·LINE_END
LINE_END -> ·SEMICOLON LF
Transitions: LINE_END->S21, SEMICOLON->S15,

State S14
Reduce items:
PORTFOLIO_EX -> PORTFOLIO_EX FREE_TEXT LINE_END · [DESCRIPTION]

State S15
Shift items:
LINE_END -> SEMICOLON ·LF
Transitions: [line break]->S22,

State S16
Reduce items:
PORTFOLIO_EX -> PORTFOLIO FREE_TEXT LINE_END · [DESCRIPTION]

State S17
Shift items:
FIRMS -> FIRMS_LIST FIRMS_VARIATIONS ·LINE_END
LINE_END -> ·SEMICOLON LF
Transitions: LINE_END->S23, SEMICOLON->S15,

State S18
Reduce items:
FIRMS_VARIATIONS -> ALL_FIRMS · [SEMICOLON]

State S19 (Inadequate)
Shift items:
FIRMS_LIST -> FIRMS_LIST ·COMMA FIRM
Reduce items:
FIRMS_VARIATIONS -> FIRMS_LIST · [SEMICOLON]
Transitions: COMMA->S24,

State S20
Reduce items:
FIRMS_LIST -> FIRM · [COMMA SEMICOLON]

State S21
Reduce items:
DESCRIPTION -> DESCRIPTION FREE_TEXT LINE_END · [FIRMS_LIST]

State S22
Reduce items:
LINE_END -> SEMICOLON LF · [EOF DESCRIPTION FIRMS_LIST]

State S23
Reduce items:
FIRMS -> FIRMS_LIST FIRMS_VARIATIONS LINE_END · [EOF]

State S24
Shift items:
FIRMS_LIST -> FIRMS_LIST COMMA ·FIRM
Transitions: FIRM->S25,

State S25
Reduce items:
FIRMS_LIST -> FIRMS_LIST COMMA FIRM · [COMMA SEMICOLON] </small>

Roman, thanks for your desire to help! )

It's possible to scan and parse hard grammars like C# or SQL with Irony.
I'm sure it have to work for my task! What can I try else?

New Post: Ambiguous FreeTextLiteral

rivantsov — Wed, 18 Nov 2009 16:40:45 GMT

It would be helpful next time if you provide a sample that you try to parse and also point to the exact place when parser reports an error.

But from what I see my guess would be that the trouble is in PORTFOLIO_EX definition - it should not have LINE_END in both clauses. Semicolon and linebreak are defined as terminators for FREE_TEXT elements and are consumed by this element - in the sense that when parser completes reading FREE_TEXT, the current position of the scanner is at the beginning of the next line, after "\n". But parser still wants to see LINE_END, because it is specified in the rule.

So just try removing LINE_END from PORTFOLIO_EX rule. Let me know if it works. If it's not the problem, then please provide a sample and failure position

Roman

New Post: Ambiguous FreeTextLiteral

DenisKolodin — Tue, 17 Nov 2009 20:09:41 GMT

I want to see a free text in the parse tree so I haven't added FreeTextLiteral to NonGrammarTerminals.
And FreeTextTerminal is always in the list of FallBackTerminals.
I have to scan and parse a text like this:

PORTFOLIO_EX Portfolio name;
DESCRIPTION Any description;
FIRMS_LIST ALL_FIRMS;

I wrote the code below for it:

          var PORTFOLIO = new NonTerminal("PORTFOLIO");
          var METADATA = new NonTerminal("METADATA");
          var PORTFOLIO_EX = new NonTerminal("PORTFOLIO_EX");
          var DESCRIPTION = new NonTerminal("DESCRIPTION");
          var FIRMS = new NonTerminal("FIRMS");
          var LINE_END = new NonTerminal("LINE_END");
          var FIRMS_VARIATIONS = new NonTerminal("FIRMS_VARIATIONS");
          var FIRMS_LIST = new NonTerminal("FIRMS_LIST");

          var FREE_TEXT = new FreeTextLiteral("FREE_TEXT", ";", "\n");
          var SEMICOLON = ToTerm(";", "SEMICOLON");
          var COMMA = ToTerm(",", "COMMA");
          var FIRM = new IdentifierTerminal("FIRM");


          this.Root = PORTFOLIO;
          PORTFOLIO.Rule = METADATA;
          METADATA.Rule = PORTFOLIO_EX + DESCRIPTION + FIRMS;
          PORTFOLIO_EX.Rule = "PORTFOLIO_EX" + FREE_TEXT + LINE_END | "PORTFOLIO" + FREE_TEXT + LINE_END;
          DESCRIPTION.Rule = "DESCRIPTION" + FREE_TEXT + LINE_END;
          FIRMS.Rule = "FIRMS_LIST" + FIRMS_VARIATIONS + LINE_END;
          FIRMS_VARIATIONS.Rule = "ALL_FIRMS" | FIRMS_LIST;
          FIRMS_LIST.Rule = MakePlusRule(FIRMS_LIST, COMMA, FIRM);
          LINE_END.Rule = SEMICOLON + NewLine;

And it's not working (

Maybe, are there defects in the rules?

New Post: Ambiguous FreeTextLiteral

rivantsov — Tue, 17 Nov 2009 17:52:00 GMT

First, do you use this FreeTextLiteral in any of the rules? if not, you should add it to NonGrammarTerminals.

Otherwise, if it is used somewhere, then Scanner should find it. If it does not define any Firsts prefixes (GetFirsts() return null or empty list), it should be automatically added to FallbackTerminals. Step thru the process and check that this is so (stop when scanner tries to find a terminal and check that your FreeTextTerminal is in the list of FallBackTerminals)

Let me know the results

Roman

New Post: Ambiguous FreeTextLiteral

DenisKolodin — Tue, 17 Nov 2009 17:45:05 GMT

Hello!
I have a problem with scanning of FreeTextLiteral.
That one was detected when input text starts with any character missing in hash table.
How to create Terminal which fits free text starts with any character (including latin characters).
Can anybody help me?

New Post: Macro expansion

rivantsov — Sat, 14 Nov 2009 16:38:55 GMT

"one token ahead", for "macro" in Identifier + macro - yes, you interpret it right, macro is not considered this one token lookahead when parser in the position before identifier, it is identifier itself. The trouble is that at this point Parser must make a decision between 2 or more alternatives, and all it sees is CurrentToken (identifier), which IS the single lookahead token.

About built-in automatic lookup facility for cases like this - I actually thought about this, and it may be possible, but I didnt' figure out details yet. I may come back to this in the future

Roman

New Post: Identifier results when grammar is case insensitive

rivantsov — Sat, 14 Nov 2009 16:33:14 GMT

The issue is fixed in the latest drop. It turned out the problem was a bit deeper that one would expect, it was actually several problems spread out in several places. Now all identifiers show up "as is", with original casing even in case-insensitive languages. It became interpreter's responsibility to handle case variations if language is case insensitive - by creating case-insesitive dictionaries for local and global variables

thanks again

Roman

New Post: Identifier results when grammar is case insensitive

MindCore — Thu, 12 Nov 2009 01:15:36 GMT

Thanks for following up.

New Post: Identifier results when grammar is case insensitive

rivantsov — Wed, 11 Nov 2009 14:44:27 GMT

I'm back to this issue, and I'm able to reproduce it. With GwBasic grammar and script like this:

1 x = xX

2 y = xX * 2

The first occurrence of xX shows as "xX" in parse tree, while the second one is "xx"

Will investigate and fix it.

Good catch! - thanks again!

New Post: Macro expansion

alexandre_mutel — Wed, 11 Nov 2009 11:00:31 GMT

rivantsov wrote:

"..Good point for FreeTextLiteral, although i still need to parse statements inside a macro... and it seems not possible (yet) to mix unstructured and structured lines in Irony? (at the condition that an unstructured line doesn't match a valid structured line...) "

Well, for me the first problem is actually define formally how the macro body should be parsed; so far I don't have a clear picture honestly. Only after that we'll try to express these rules in Irony.

My fault, you are right. Trying to quickly prototype without giving you a kind of "whole picture" and asking why is not working with Irony is not fair! Moreover, i'm discovering that the macro body is only parsed when expansion occurs. It means that you were right when you advise me to use FreeTextLiteral.

ASM parsing is simple in many ways but assemblers have also a lots of weird syntax behaviour... instead of working on a MASM syntax, i should more probably develop an easier syntax to parse... i'll see that option too.

rivantsov wrote:

First of all Antlr is build on quite different algorithmic foundation, it uses top-down (LL) parser, while Irony follows bottom-up (LALR) algorithm. So out-of-the-box abilities of the algorithms are different. Generally, LALR algorithms are considered less restrictive, faster and overall preferable to LL. As one example, LALR has not problem with Left-recursive rules, while LL cannot handle them at all, you have to refactor the grammar to get rid of these. As for using lookaheads, both approaches handle a single lookahead token as a facility of algorithm itself. Bigger lookaheads if needed are usually handled as custom code hooked into Parser that kicks in at point of indecision, looks ahead in token stream and gives parser an advice. There is such facility in Irony, look at c# sample. there is a custom lookahead code there that helps parser decide what is "<" in input - comparison operator or opening bracket for type parameter. In your case very similar code can do the same for looking up "macro" keyword in the stream.

Anyway, good luck with your research, let me know if you need anything else

Thanks! I missed the ResolveCode() in C# grammar. I have tested in the asm grammar and it's working great, even with unstructured token lines. So this ResolveCode() + On OnResolvingConflict() resolves the thing!

About LALR(1) and the "one token ahead", you are saying that "macro" in "Identifier + macro" is not considered as a one token ahead of "Identifier"? Or the token ahead is counting the current token? (sorry for this dummy question, I'm not played with parsing for a while!)

The ResolveCode() + OnResolvingConflict() is really helpful and easy to use but i'm wondering if it would be relevant for Irony to provide a builtin simple disambiguation for one token ahead (or two, if the one token ahead is the current token)? But ok, that's not a priority. Hope that you will be able to push a stable 1.0 before the end of the year.