org.apache.lucene.analysis

Class Tokenizer

java.lang.Object

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.Tokenizer

Direct Known Subclasses:

CharTokenizer, KeywordTokenizer, SinkTokenizer, StandardTokenizer

public abstract class Tokenizer
extends TokenStream

A Tokenizer is a TokenStream whose input is a Reader.

This is an abstract class.

NOTE: subclasses must override at least one of TokenStream.next() or TokenStream.next(Token).

NOTE: subclasses overriding TokenStream.next(Token) must call Token.clear().

Method Summary

Modifier and Type	Method and Description
void	close() By default, closes the input Reader.
void	reset(java.io.Reader input) Expert: Reset the tokenizer to a new reader.

Methods inherited from class org.apache.lucene.analysis.TokenStream

next, next, reset

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

close

public void close()
           throws java.io.IOException

By default, closes the input Reader.

Overrides:

close in class TokenStream

Throws:

java.io.IOException

reset

public void reset(java.io.Reader input)
           throws java.io.IOException

Expert: Reset the tokenizer to a new reader. Typically, an analyzer (in its reusableTokenStream method) will use this to re-use a previously created tokenizer.

Throws:

java.io.IOException