com.singularsys.jep.configurableparser
Class Tokenizer

java.lang.Object
  extended by com.singularsys.jep.configurableparser.Tokenizer

public class Tokenizer
extends java.lang.Object

Breaks the input into a set of tokens.


Field Summary
protected  java.io.BufferedReader br
           
protected  java.lang.String currentLine
           
protected  int currentLineNumber
           
protected  int currentPos
           
static TerminatorToken EOF
          A token which represents the end of file/input.
static TerminatorToken EOL
          A token which represents and end of line.
protected  java.util.List<TokenMatcher> matchers
           
protected  java.util.List<Token> tokens
           
 
Constructor Summary
Tokenizer(java.io.BufferedReader br, java.util.List<TokenMatcher> matchers)
          Construct a tokenizer for a given input.
 
Method Summary
 java.util.List<Token> scan()
          Scans the input, breaking it down into tokens.
static java.lang.String toString(java.util.List<Token> toks)
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

matchers

protected java.util.List<TokenMatcher> matchers

currentPos

protected int currentPos

currentLine

protected java.lang.String currentLine

currentLineNumber

protected int currentLineNumber

br

protected java.io.BufferedReader br

tokens

protected java.util.List<Token> tokens

EOL

public static TerminatorToken EOL
A token which represents and end of line.


EOF

public static TerminatorToken EOF
A token which represents the end of file/input. A TokenMatcher can return this symbol if it wishes to terminate processing of the input.

Constructor Detail

Tokenizer

public Tokenizer(java.io.BufferedReader br,
                 java.util.List<TokenMatcher> matchers)
Construct a tokenizer for a given input.

Parameters:
br - the input to read from.
matchers - a set of TokenMatchers to match the given input.
Method Detail

scan

public java.util.List<Token> scan()
                           throws ParseException
Scans the input, breaking it down into tokens. Returns null if attempts to read past end of the file

Returns:
the list of tokens that the input was broken down into
Throws:
ParseException

toString

public static java.lang.String toString(java.util.List<Token> toks)


Copyright © 2008 Singular Systems http://www.singularsys.com/jep