com.datumbox.framework.core.common.text.tokenizers

Class AbstractTokenizer

java.lang.Object

com.datumbox.framework.core.common.text.tokenizers.AbstractTokenizer

Direct Known Subclasses:

WhitespaceTokenizer

public abstract class AbstractTokenizer
extends java.lang.Object

This abstract class should be implemented by classes that are responsible to get a string as input and separate it into tokens/keywords.

Constructor Summary

Constructors

Constructor and Description
AbstractTokenizer()

Method Summary

Modifier and Type	Method and Description
abstract java.util.List<java.lang.String>	tokenize(java.lang.String text) The tokenize method accepts a string as input and returns a list of tokens.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AbstractTokenizer

public AbstractTokenizer()

Method Detail

tokenize

public abstract java.util.List<java.lang.String> tokenize(java.lang.String text)

The tokenize method accepts a string as input and returns a list of tokens.

Parameters:

text -

Returns: