jwo.utils.expression
Class Tokenizer

java.lang.Object
  jwo.utils.expression.Tokenizer

public class Tokenizer
extends Object

A tokenizer that uses regular expressions to parse text. Modified from original code described at www.devx.com/premier/mgznarch/javapro/2001/05may01/ps0105/ps0105.asp.

Version:

2.3, 15th February, 2001, modified 23rd June, 2006.

Author:

Daniel Savarese, modified by Jo Wood.

Field Summary

static String

DEFAULT_SEPARATOR Default separator used to distiguish tokens.

Constructor Summary

Tokenizer()
Creates a default tokenizer with no initial expression to process.

Tokenizer(String expressionText)
Creates a tokenizer using the given initial expression.

Method Summary

void	addToken(String lexeme, Object token) Adds a token to the currently stored regular expression to use for processing.
Token	nextToken() Returns the next token in the input.
void	reset() Resets the list of tokens to process.
void	setInput(String expressionText) Sets the input sequence of characters to process.
void	setSeparator(String separatorText) Sets the separator used to distinguish between tokens.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_SEPARATOR

public static final String DEFAULT_SEPARATOR

Default separator used to distiguish tokens.

See Also:

Constant Field Values

Constructor Detail

Tokenizer

public Tokenizer()

Creates a default tokenizer with no initial expression to process.

Tokenizer

public Tokenizer(String expressionText)

Creates a tokenizer using the given initial expression.

Parameters:

expressionText - Expression to process.

Method Detail

setInput

public void setInput(String expressionText)

Sets the input sequence of characters to process.

Parameters:

expressionText - Expression to process.

setSeparator

public void setSeparator(String separatorText)
                  throws PatternSyntaxException

Sets the separator used to distinguish between tokens. The default is any whitespace characters ("\\s+").

Parameters:

separatorText - Regular expression identifying token separators.

Throws:

PatternSyntaxException - If problem compiling the regular expression.

addToken

public void addToken(String lexeme,
                     Object token)
              throws PatternSyntaxException

Adds a token to the currently stored regular expression to use for processing.

Parameters:

lexeme - The expression lexeme to add.

token - The type of token to add.

Throws:

PatternSyntaxException - If problem compiling the lexeme as a regular expression.

reset

public void reset()

Resets the list of tokens to process.

nextToken

public Token nextToken()
                throws TokenNotFoundException

Returns the next token in the input. If at end of input returns null.

Returns:

Next token read or null if at end of input.

Throws:

TokenNotFoundException - If unknown token is found.