jsat.text

Class HashedTextDataLoader

java.lang.Object

jsat.text.HashedTextDataLoader

All Implemented Interfaces:

java.io.Serializable, TextVectorCreator

Direct Known Subclasses:

ClassificationHashedTextDataLoader

public abstract class HashedTextDataLoader
extends java.lang.Object
implements TextVectorCreator

This class provides a framework for loading datasets made of Text documents as hashed feature vectors. Text is broken up into a sequence of tokens using a Tokenizer, that must be provided. The weights used will be determined by some word weighting scheme.
The user adds documents to the initial dataset using the addOriginalDocument(java.lang.String) method. The finishAdding() must be called when no more documents are left to add, at which point class will take care of calling the WordWeighting.setWeight(java.util.List, java.util.List) method to configure the word weighting used with the original data added.

After the initial dataset is loaded, new strings can be converted to vectors using the newText(java.lang.String) method. This should only be called after finishAdding().

Instance of this class will keep a reference to all originally added vectors. To transform new texts into vectors without keeping references to all of the original vectors, the getTextVectorCreator() will return an object that perform the transformation.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
HashedTextDataLoader(int dimensionSize, Tokenizer tokenizer, WordWeighting weighting)
HashedTextDataLoader(Tokenizer tokenizer, WordWeighting weighting)

Method Summary

Modifier and Type	Method and Description
DataSet	getDataSet() Returns a new data set containing the original data points that were loaded with this loader.
TextVectorCreator	getTextVectorCreator() Returns the TextVectorCreator used by this data loader to convert documents into vectors.
Vec	newText(java.lang.String input) Converts the given input text into a vector representation.
Vec	newText(java.lang.String input, java.lang.StringBuilder workSpace, java.util.List<java.lang.String> storageSpace) Converts the given input text into a vector representation

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HashedTextDataLoader

public HashedTextDataLoader(Tokenizer tokenizer,
                            WordWeighting weighting)

HashedTextDataLoader

public HashedTextDataLoader(int dimensionSize,
                            Tokenizer tokenizer,
                            WordWeighting weighting)

Method Detail

getDataSet

public DataSet getDataSet()

Returns a new data set containing the original data points that were loaded with this loader.

Returns:

an appropriate data set for this loader

newText

public Vec newText(java.lang.String input)

Description copied from interface: TextVectorCreator

Converts the given input text into a vector representation.

Specified by:

newText in interface TextVectorCreator

Parameters:

input - the input string

Returns:

a vector representation

newText

public Vec newText(java.lang.String input,
                   java.lang.StringBuilder workSpace,
                   java.util.List<java.lang.String> storageSpace)

Description copied from interface: TextVectorCreator

Converts the given input text into a vector representation

Specified by:

newText in interface TextVectorCreator

Parameters:

input - the input string

workSpace - an already allocated (but empty) string builder than can be used as a temporary work space.

storageSpace - an already allocated (but empty) list to place the tokens into

Returns:

a vector representation

getTextVectorCreator

public TextVectorCreator getTextVectorCreator()

Returns the TextVectorCreator used by this data loader to convert documents into vectors.

Returns:

the text vector creator used by this class