jsat.text.tokenizer

Interface Tokenizer

All Superinterfaces:

java.io.Serializable

All Known Implementing Classes:

NaiveTokenizer, NGramTokenizer, StemmingTokenizer, StopWordTokenizer

public interface Tokenizer
extends java.io.Serializable

Interface for taking the text of a document and breaking it up into features. For example "This doc" might become "this" and "doc"

Method Summary

Modifier and Type	Method and Description
java.util.List<java.lang.String>	tokenize(java.lang.String input) Breaks the input string into a series of tokens that may be used as features for a classifier.
void	tokenize(java.lang.String input, java.lang.StringBuilder workSpace, java.util.List<java.lang.String> storageSpace) Breaks the input string into a series of tokens that may be used as features for a classifier.

Method Detail

tokenize

java.util.List<java.lang.String> tokenize(java.lang.String input)

Breaks the input string into a series of tokens that may be used as features for a classifier. The returned tokens must be either new string objects or interned strings. If a token is returned that is backed by the original document, memory may get leaked by processes consuming the token.
This method should be thread safe

Parameters:

input - the string to tokenize

Returns:

an already allocated list to place the tokens into

tokenize

void tokenize(java.lang.String input,
              java.lang.StringBuilder workSpace,
              java.util.List<java.lang.String> storageSpace)

Breaks the input string into a series of tokens that may be used as features for a classifier. The returned tokens must be either new string objects or interned strings. If a token is returned that is backed by the original document, memory may get leaked by processes consuming the token.
This method should be thread safe

Parameters:

input - the string to tokenize

workSpace - an already allocated (but empty) string builder than can be used as a temporary work space.

storageSpace - an already allocated (but empty) list to place the tokens into