This project has moved and is read-only. For the latest updates, please go here.

Adding tokens in TokenFilter

Feb 9, 2015 at 5:26 PM
I have a custom TokenFilter and I'm trying to figure out how to correctly add new tokens in the BeginFiltering method. KeyTerms appear to work properly because I can grab the actual KeyTerm that I want from the grammar and everyone is happy.

What doesn't work is if I have an IdentifierTerminal named "Term". If I want to add a new "Term" in my TokenFilter I can't just create a new IdentifierTerminal in the BeginFiltering method because the state machine in the parser only knows about the "Term" that was created in my grammar definition.

Any new "Term" that I create won't cause a hit in the currState.Actions hashmap. Am I attempting to do something that is not allowed, or is there a legitimate way to create new tokens like this in a TokenFilter that won't cause the state machine to stop?

Thanks for any help.
Feb 9, 2015 at 5:52 PM
I think that I fixed this myself. The terminal used still needs to be the one specified in the grammar. The token contains that terminal as well as the text associated with that token. Created a new token using the terminal in the grammar object and that seems to work.