jsat.io

Class JSATData

java.lang.Object

jsat.io.JSATData

public class JSATData
extends java.lang.Object

JSAT Data Loader provides a simple binary file format for storing and reading datasets. All datasets can always be read back in as a SimpleDataSet, and ClassificationDataSet and RegressionDataSet datasets can be read back in as their original types.

For well behaved datasets (where most numeric features are integer values), an uncompressed JSAT Dataset file may be larger than a similar ARFF or LIBSVM file. This is because JSAT always uses 32 or 64 bits (4 or 8 bytes) for every value, where values stored as a string could use as little as 1 or 2 bytes for simple values. However, JSAT's storage will be consistent - data which uses the floating point values (such as the number "0.25098039215686274") will use additional bytes in the human readable ARFF and LIBSVM formats that are not necessary, where the binary JSAT format will stay the same size.

Even when JSAT produces larger files, since it is a simple binary format, reading and writing will usually be significantly faster.
Additional storage savings can be obtained by using the GZIPOutputStream when storing a dataset, and then decompressed when read back in using GZIPInputStream.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	JSATData.DatasetTypeMarker
static class	JSATData.FloatStorageMethod

Field Summary

Fields

Modifier and Type	Field and Description
static byte[]	MAGIC_NUMBER
static byte	STRING_ENCODING_ASCII
static byte	STRING_ENCODING_UTF_16

Method Summary

Modifier and Type	Method and Description
static DataWriter	getWriter(java.io.OutputStream out, CategoricalData[] catInfo, int dim, CategoricalData predicting, JSATData.FloatStorageMethod fpStore, DataWriter.DataSetType type) Returns a DataWriter object which can be used to stream a set of arbitrary datapoints into the given output stream.
static DataSet<?>	load(java.io.InputStream inRaw) This loads a JSAT dataset from an input stream, and will not do any of its own buffering.
static ClassificationDataSet	loadClassification(java.io.InputStream inRaw) Loads in a JSAT dataset as a ClassificationDataSet.
static RegressionDataSet	loadRegression(java.io.InputStream inRaw) Loads in a JSAT dataset as a RegressionDataSet.
static SimpleDataSet	loadSimple(java.io.InputStream inRaw) Loads in a JSAT dataset as a SimpleDataSet.
static <Type extends DataSet<Type>> void	writeData(DataSet<Type> dataset, java.io.OutputStream outRaw) This method writes out a JSAT dataset to a binary format that can be read in again later, and could be read in other languages. The format that is used will understand both ClassificationDataSet and RegressionDataSet datasets as special cases, and will store the target values in the binary file.
static <Type extends DataSet<Type>> void	writeData(DataSet<Type> dataset, java.io.OutputStream outRaw, JSATData.FloatStorageMethod fpStore) This method writes out a JSAT dataset to a binary format that can be read in again later, and could be read in other languages. The format that is used will understand both ClassificationDataSet and RegressionDataSet datasets as special cases, and will store the target values in the binary file.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAGIC_NUMBER

public static final byte[] MAGIC_NUMBER

STRING_ENCODING_ASCII

public static final byte STRING_ENCODING_ASCII

See Also:

Constant Field Values

STRING_ENCODING_UTF_16

public static final byte STRING_ENCODING_UTF_16

See Also:

Constant Field Values

Method Detail

writeData

public static <Type extends DataSet<Type>> void writeData(DataSet<Type> dataset,
                                                          java.io.OutputStream outRaw)
                                                   throws java.io.IOException

This method writes out a JSAT dataset to a binary format that can be read in again later, and could be read in other languages.

The format that is used will understand both ClassificationDataSet and RegressionDataSet datasets as special cases, and will store the target values in the binary file. When read back in, they can be returned as their original dataset type, or treated as normal fields as a SimpleDataSet.
The storage format chosen for floating point values will chose a method that results in no loss of precision when reading the data back in.

Type Parameters:

Type -

Parameters:

dataset - the dataset to write out to a binary file

outRaw - the raw output stream, the caller should provide a buffered stream.

Throws:

java.io.IOException

writeData

public static <Type extends DataSet<Type>> void writeData(DataSet<Type> dataset,
                                                          java.io.OutputStream outRaw,
                                                          JSATData.FloatStorageMethod fpStore)
                                                   throws java.io.IOException

This method writes out a JSAT dataset to a binary format that can be read in again later, and could be read in other languages.

The format that is used will understand both ClassificationDataSet and RegressionDataSet datasets as special cases, and will store the target values in the binary file. When read back in, they can be returned as their original dataset type, or treated as normal fields as a SimpleDataSet.

Type Parameters:

Type -

Parameters:

dataset - the dataset to write out to a binary file

outRaw - the raw output stream, the caller should provide a buffered stream.

fpStore - the storage method of storing floating point values, which may result in a loss of precision depending on the method chosen.

Throws:

java.io.IOException

getWriter

public static DataWriter getWriter(java.io.OutputStream out,
                                   CategoricalData[] catInfo,
                                   int dim,
                                   CategoricalData predicting,
                                   JSATData.FloatStorageMethod fpStore,
                                   DataWriter.DataSetType type)
                            throws java.io.IOException

Returns a DataWriter object which can be used to stream a set of arbitrary datapoints into the given output stream. This works in a thread safe manner.

Parameters:

out - the location to store all the data

catInfo - information about the categorical features to be written

dim - information on how many numeric features exist

predicting - information on the class label, may be null if not a classification dataset

fpStore - the format floating point values should be stored as

type - what type of data set (simple, classification, regression) to be written

Returns:

the DataWriter that the actual points can be streamed through

Throws:

java.io.IOException

load

public static DataSet<?> load(java.io.InputStream inRaw)
                       throws java.io.IOException

This loads a JSAT dataset from an input stream, and will not do any of its own buffering. The DataSet will be returned as either a SimpleDataSet, ClassificationDataSet, or RegressionDataSet depending on what type of dataset was originally written out.

Type Parameters:

Type -

Parameters:

inRaw - the input stream, caller should buffer it

Returns:

a dataset

Throws:

java.io.IOException

loadSimple

public static SimpleDataSet loadSimple(java.io.InputStream inRaw)
                                throws java.io.IOException

Loads in a JSAT dataset as a SimpleDataSet. So long as the input stream is valid, this will not fail.

Parameters:

inRaw - the input stream, caller should buffer it

Returns:

a SimpleDataSet object

Throws:

java.io.IOException

loadClassification

public static ClassificationDataSet loadClassification(java.io.InputStream inRaw)
                                                throws java.io.IOException

Loads in a JSAT dataset as a ClassificationDataSet. An exception will be thrown if the original dataset in the file was not a ClassificationDataSet.

Parameters:

inRaw - the input stream, caller should buffer it

Returns:

a ClassificationDataSet object

Throws:

java.io.IOException

java.lang.ClassCastException - if the original dataset was a not a ClassificationDataSet

loadRegression

public static RegressionDataSet loadRegression(java.io.InputStream inRaw)
                                        throws java.io.IOException

Loads in a JSAT dataset as a RegressionDataSet. An exception will be thrown if the original dataset in the file was not a RegressionDataSet.

Parameters:

inRaw - the input stream, caller should buffer it

Returns:

a RegressionDataSet object

Throws:

java.io.IOException

java.lang.ClassCastException - if the original dataset was a not a RegressionDataSet