andyr.jtokeniser
Class Tokeniser

java.lang.Object
  andyr.jtokeniser.Tokeniser

Direct Known Subclasses:

BreakIteratorTokeniser, RegexSeparatorTokeniser, RegexTokeniser, SentenceTokeniser, StringTokeniser, WhiteSpaceTokeniser

public abstract class Tokeniser
extends java.lang.Object

Tokeniser is an abstract base class for a variety of Tokenisers.

Version:

1.2 (01-Aug-2005)

Author:

Andrew Roberts

Field Summary

protected int	currentTokenPosition
protected java.util.List	tokens

Constructor Summary

protected

Tokeniser()

Method Summary

int	countTokens() Calculates the number of times that this tokeniser's nextToken method can be called before it generates an exception.
java.util.List	getTokens() Returns a List of string tokens.
boolean	hasMoreTokens() Tests if there are more tokens available from this tokeniser's string.
java.lang.String	nextToken() Returns the next token from this tokeniser.
int	numberOfTokens() Returns the total number of tokens that were extracted when tokenising the input.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

tokens

protected java.util.List tokens

currentTokenPosition

protected int currentTokenPosition

Constructor Detail

Tokeniser

protected Tokeniser()

Method Detail

countTokens

public int countTokens()

Calculates the number of times that this tokeniser's nextToken method can be called before it generates an exception.

Returns:

the number of tokens remaining in the string.

numberOfTokens

public int numberOfTokens()

Returns the total number of tokens that were extracted when tokenising the input.

Returns:

the number of tokens in the string

hasMoreTokens

public boolean hasMoreTokens()

Tests if there are more tokens available from this tokeniser's string. If this method returns true, then a subsequent call to nextToken will successfully return a token.

Returns:

true if and only if there is at least one token in the string after the current position; false otherwise.

nextToken

public java.lang.String nextToken()
                           throws java.util.NoSuchElementException

Returns the next token from this tokeniser.

Returns:

the next token from this tokeniser.

Throws:

java.util.NoSuchElementException - if there are no more tokens in this tokeniser's string.

getTokens

public java.util.List getTokens()

Returns a List of string tokens.

Returns:

a List of string tokens.