Package com.singularsys.jep.configurableparser

Class Tokenizer

java.lang.Object

com.singularsys.jep.configurableparser.Tokenizer

public class Tokenizer
extends Object

Breaks the input into a set of tokens.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	Tokenizer.StandardTokenizerFactory	Factory creating new instances of the tokenizer.

Field Summary

Fields

Modifier and Type	Field	Description
protected final BufferedReader	br
protected String	currentLine
protected int	currentLineNumber
protected int	currentPos
protected final List<TokenMatcher>	matchers
protected List<Token>	tokens

Constructor Summary

Constructors

Constructor	Description
Tokenizer(BufferedReader br, ConfigurableParser cp)	Construct a tokenizer for a given input.

Method Summary

Modifier and Type	Method	Description
List<Token>	scan()	Scans the input, breaking it down into tokens.
static String	toString(List<Token> toks)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

matchers

protected final List<TokenMatcher> matchers

currentPos

protected int currentPos

currentLine

protected String currentLine

currentLineNumber

protected int currentLineNumber

br

protected final BufferedReader br

tokens

protected List<Token> tokens

Constructor Details

Tokenizer

public Tokenizer(BufferedReader br,
 ConfigurableParser cp)

Construct a tokenizer for a given input.

Parameters:

br - the input to read from.

cp - The configurable parser instance.

Since:

3.4.0

Method Details

scan

public List<Token> scan()
                 throws ParseException

Scans the input, breaking it down into tokens.

Returns:

the list of tokens that the input was broken down into, ";;" returns an empty list, null if attempts to read past end of the file

Throws:

ParseException - if there is a syntax error in the input

toString

public static String toString(List<Token> toks)