jwo.utils.expression
Class Tokenizer

java.lang.Object
  extended byjwo.utils.expression.Tokenizer

public class Tokenizer
extends Object

A tokenizer that uses regular expressions to parse text. Modified from original code described at www.devx.com/premier/mgznarch/javapro/2001/05may01/ps0105/ps0105.asp.

Version:
2.2, 15th February, 2001, modified 18th July, 2003.
Author:
Daniel Savarese, modified by Jo Wood.

Field Summary
static String DEFAULT_SEPARATOR
           
 
Constructor Summary
Tokenizer()
          Creates a default tokenizer with no initial expression to process.
Tokenizer(String expressionText)
          Creates a tokenizer using the given initial expression.
 
Method Summary
 void addToken(String lexeme, Object token)
          Adds a token to the currently stored regular expression to use for processing.
 Token nextToken()
          Returns the next token in the input.
 void reset()
          Resets the list of tokens to process.
 void setInput(String expressionText)
          Sets the input sequence of characters to process.
 void setSeparator(String separatorText)
          Sets the separator used to distinguish between tokens.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

DEFAULT_SEPARATOR

public static final String DEFAULT_SEPARATOR
See Also:
Constant Field Values
Constructor Detail

Tokenizer

public Tokenizer()
Creates a default tokenizer with no initial expression to process.


Tokenizer

public Tokenizer(String expressionText)
Creates a tokenizer using the given initial expression.

Parameters:
expressionText - Expression to process.
Method Detail

setInput

public void setInput(String expressionText)
              throws PatternSyntaxException
Sets the input sequence of characters to process.

Parameters:
expressionText - Expression to process.
Throws:
PatternSyntaxException

setSeparator

public void setSeparator(String separatorText)
                  throws PatternSyntaxException
Sets the separator used to distinguish between tokens. The default is any whitespace characters ("\\s+").

Parameters:
separatorText - Regular expression identifying token separators.
Throws:
PatternSyntaxException

addToken

public void addToken(String lexeme,
                     Object token)
              throws PatternSyntaxException
Adds a token to the currently stored regular expression to use for processing.

Parameters:
lexeme - The expression lexeme to add.
token - The type of token to add.
Throws:
PatternSyntaxException

reset

public void reset()
Resets the list of tokens to process.


nextToken

public Token nextToken()
                throws TokenNotFoundException
Returns the next token in the input. If at end of input returns null. If no token is found, throws an exception.

Throws:
TokenNotFoundException


Copyright Jo Wood, 1996-2005, last modified, 11th March, 2005