|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectcom.singularsys.jep.configurableparser.Tokenizer
public class Tokenizer
Breaks the input into a set of tokens.
Field Summary | |
---|---|
protected java.io.BufferedReader |
br
|
protected java.lang.String |
currentLine
|
protected int |
currentLineNumber
|
protected int |
currentPos
|
static TerminatorToken |
EOF
A token which represents the end of file/input. |
static TerminatorToken |
EOL
A token which represents and end of line. |
protected java.util.List<TokenMatcher> |
matchers
|
protected java.util.List<Token> |
tokens
|
Constructor Summary | |
---|---|
Tokenizer(java.io.BufferedReader br,
java.util.List<TokenMatcher> matchers)
Construct a tokenizer for a given input. |
Method Summary | |
---|---|
java.util.List<Token> |
scan()
Scans the input, breaking it down into tokens. |
static java.lang.String |
toString(java.util.List<Token> toks)
|
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
---|
protected java.util.List<TokenMatcher> matchers
protected int currentPos
protected java.lang.String currentLine
protected int currentLineNumber
protected java.io.BufferedReader br
protected java.util.List<Token> tokens
public static TerminatorToken EOL
public static TerminatorToken EOF
Constructor Detail |
---|
public Tokenizer(java.io.BufferedReader br, java.util.List<TokenMatcher> matchers)
br
- the input to read from.matchers
- a set of TokenMatchers to match the given input.Method Detail |
---|
public java.util.List<Token> scan() throws ParseException
ParseException
public static java.lang.String toString(java.util.List<Token> toks)
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |