Farkle


Tokenizer Type

A class that breaks down the characters of a CharStream into Tokens.

User code can inherit this class and implement additional tokenizer logic by overriding the GetNextToken method.

Constructors

Constructor Description

Tokenizer()

Full Usage: Tokenizer()

Returns: Tokenizer
Returns: Tokenizer

Instance members

Instance member Description

this.GetNextToken

Full Usage: this.GetNextToken

Parameters:
    transformer : ITransformer<Terminal> - This parameter is used for the post-processor. It should be passed to the base method if called.
    input : CharStream - The CharStream whose characters will be processed.

Returns: Token
Modifiers: abstract

Gets the next Token from a CharStream.

transformer : ITransformer<Terminal>

This parameter is used for the post-processor. It should be passed to the base method if called.

input : CharStream

The CharStream whose characters will be processed.

Returns: Token