jsat.text.tokenizer

Class StemmingTokenizer

java.lang.Object

jsat.text.tokenizer.StemmingTokenizer

All Implemented Interfaces:

java.io.Serializable, Tokenizer

public class StemmingTokenizer
extends java.lang.Object
implements Tokenizer

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
StemmingTokenizer(Stemmer stemmer, Tokenizer baseTokenizer)

Method Summary

Modifier and Type	Method and Description
java.util.List<java.lang.String>	tokenize(java.lang.String input) Breaks the input string into a series of tokens that may be used as features for a classifier.
void	tokenize(java.lang.String input, java.lang.StringBuilder workSpace, java.util.List<java.lang.String> storageSpace) Breaks the input string into a series of tokens that may be used as features for a classifier.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StemmingTokenizer

public StemmingTokenizer(Stemmer stemmer,
                         Tokenizer baseTokenizer)

Method Detail

tokenize

public java.util.List<java.lang.String> tokenize(java.lang.String input)

Description copied from interface: Tokenizer

Breaks the input string into a series of tokens that may be used as features for a classifier. The returned tokens must be either new string objects or interned strings. If a token is returned that is backed by the original document, memory may get leaked by processes consuming the token.
This method should be thread safe

Specified by:

tokenize in interface Tokenizer

Parameters:

input - the string to tokenize

Returns:

an already allocated list to place the tokens into

tokenize

public void tokenize(java.lang.String input,
                     java.lang.StringBuilder workSpace,
                     java.util.List<java.lang.String> storageSpace)

Description copied from interface: Tokenizer

Breaks the input string into a series of tokens that may be used as features for a classifier. The returned tokens must be either new string objects or interned strings. If a token is returned that is backed by the original document, memory may get leaked by processes consuming the token.
This method should be thread safe

Specified by:

tokenize in interface Tokenizer

Parameters:

input - the string to tokenize

workSpace - an already allocated (but empty) string builder than can be used as a temporary work space.

storageSpace - an already allocated (but empty) list to place the tokens into