The Tokenizer type exposes the following members.
Constructors
| Name | Description | |
|---|---|---|
|
Tokenizer |
Construct a tokenizer for a given input.
|
Methods
| Name | Description | |
|---|---|---|
|
|
Equals | (Inherited from Object.) |
|
Finalize |
Allows an Object to attempt to free resources and perform other cleanup operations before the Object is reclaimed by garbage collection.
(Inherited from Object.) |
|
|
GetHashCode |
Serves as a hash function for a particular type.
(Inherited from Object.) |
|
|
GetType |
Gets the Type of the current instance.
(Inherited from Object.) |
|
|
MemberwiseClone |
Creates a shallow copy of the current Object.
(Inherited from Object.) |
|
|
Scan |
Scans the input, breaking it down into tokens.
Returns null if attempts to read past end of the file
|
|
ToString | Overloaded. |
Fields
| Name | Description | |
|---|---|---|
|
currentLine | |
|
|
currentLineNumber | |
|
|
currentPos | |
|
EOF |
A token which represents the end of file/input.
A TokenMatcher can return this symbol if it wishes to terminate processing of the input.
|
|
|
EOL |
A token which represents and end of line.
|
|
|
matchers | |
|
|
sr | |
|
|
tokens |
See Also
