Tokenizer (Farkle)

Tokenizer Type

Namespace: Farkle.Parser
Assembly: Farkle.dll
Base Type: obj

A class that breaks down the characters of a CharStream into Tokens.

User code can inherit this class and implement additional tokenizer logic by overriding the GetNextToken method.

Constructors

Constructor	Description
`Tokenizer()` Full Usage: `Tokenizer()` Returns: `Tokenizer`	Returns: `Tokenizer`

Instance members

Instance member	Description
`this.GetNextToken` Full Usage: `this.GetNextToken` Parameters: transformer : `ITransformer<Terminal>` - This parameter is used for the post-processor. It should be passed to the base method if called. input : `CharStream` - The CharStream whose characters will be processed. Returns: `Token` Modifiers: abstract	Gets the next Token from a CharStream. transformer : `ITransformer<Terminal>` This parameter is used for the post-processor. It should be passed to the base method if called. input : `CharStream` The CharStream whose characters will be processed. Returns: `Token`