jsat.regression

Class RegressionModelEvaluation

java.lang.Object

jsat.regression.RegressionModelEvaluation

public class RegressionModelEvaluation
extends java.lang.Object

Provides a mechanism to quickly evaluate a regression model on a data set. This can be done by cross validation or with a separate testing set.

Constructor Summary

Constructors

Constructor and Description
RegressionModelEvaluation(Regressor regressor, RegressionDataSet dataSet) Creates a new RegressionModelEvaluation that will perform serial training
RegressionModelEvaluation(Regressor regressor, RegressionDataSet dataSet, boolean parallel) Creates a new RegressionModelEvaluation that will perform parallel training.

Method Summary

Modifier and Type	Method and Description
void	addScorer(RegressionScore scorer) Adds a new score object that will be used as part of the evaluation when calling evaluateCrossValidation(int, java.util.Random) or evaluateTestSet(jsat.regression.RegressionDataSet).
void	evaluateCrossValidation(int folds) Performs an evaluation of the regressor using the training data set.
void	evaluateCrossValidation(int folds, java.util.Random rand) Performs an evaluation of the regressor using the training data set.
void	evaluateCrossValidation(java.util.List<RegressionDataSet> lcds) Performs an evaluation of the regressor using the training data set, where the folds of the training data set are provided by the user.
void	evaluateCrossValidation(java.util.List<RegressionDataSet> lcds, java.util.List<RegressionDataSet> trainCombinations) Note: Most people should never need to call this method.
void	evaluateTestSet(RegressionDataSet testSet) Performs an evaluation of the regressor using the initial data set to train, and testing on the given data set.
double	getErrorStndDev() Returns the standard deviation of the error from all runs
Regressor[]	getKeptModels() Returns the models that were kept after the last evaluation.
double	getMaxError() Returns the maximum squared error observed from all runs.
double	getMeanError() Returns the mean squared error from all runs.
double	getMinError() Returns the minimum squared error from all runs.
Regressor	getRegressor() Returns the regressor that was to be evaluated
OnLineStatistics	getScoreStats(RegressionScore score) Gets the statistics associated with the given score.
long	getTotalClassificationTime() Returns the total number of milliseconds spent performing regression on the testing set.
long	getTotalTrainingTime() Returns the total number of milliseconds spent training the regressor.
boolean	isKeepModels() This will keep the models trained when evaluating the model.
void	prettyPrintRegressionScores() Prints out the classification information in a convenient format.
void	setDataTransformProcess(DataTransformProcess dtp) Sets the data transform process to use when performing cross validation.
void	setKeepModels(boolean keepModels) Set this to true in order to keep the trained models after evaluation.
void	setWarmModels(Regressor... warmModels) Sets the models that will be used for warm starting training.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RegressionModelEvaluation

public RegressionModelEvaluation(Regressor regressor,
                                 RegressionDataSet dataSet,
                                 boolean parallel)

Creates a new RegressionModelEvaluation that will perform parallel training.

Parameters:

regressor - the regressor model to evaluate

dataSet - the data set to train or perform cross validation from

parallel - true if the training should be done using multiple-cores, false for single threaded.

RegressionModelEvaluation

public RegressionModelEvaluation(Regressor regressor,
                                 RegressionDataSet dataSet)

Creates a new RegressionModelEvaluation that will perform serial training

Parameters:

regressor - the regressor model to evaluate

dataSet - the data set to train or perform cross validation from

Method Detail

setKeepModels

public void setKeepModels(boolean keepModels)

Set this to true in order to keep the trained models after evaluation. They can then be retrieved used the getKeptModels() methods. The default value is false.

Parameters:

keepModels - true to keep the trained models after evaluation, false to discard them.

isKeepModels

public boolean isKeepModels()

This will keep the models trained when evaluating the model. The models can be obtained after an evaluation from getKeptModels().

Returns:

true if trained models will be kept after evaluation.

getKeptModels

public Regressor[] getKeptModels()

Returns the models that were kept after the last evaluation. null will be returned instead if isKeepModels() returns false, which is the default.

Returns:

the models that were kept after the last evaluation. Or null if if models are not being kept.

setWarmModels

public void setWarmModels(Regressor... warmModels)

Sets the models that will be used for warm starting training. If using cross-validation, the number of models given should match the number of folds. If using a test set, only one model should be given.

Parameters:

warmModels - the models to use for warm start training

setDataTransformProcess

public void setDataTransformProcess(DataTransformProcess dtp)

Sets the data transform process to use when performing cross validation. By default, no transforms are applied

Parameters:

dtp - the transformation process to clone for use during evaluation

evaluateCrossValidation

public void evaluateCrossValidation(int folds)

Performs an evaluation of the regressor using the training data set. The evaluation is done by performing cross validation.

Parameters:

folds - the number of folds for cross validation

Throws:

UntrainedModelException - if the number of folds given is less than 2

evaluateCrossValidation

public void evaluateCrossValidation(int folds,
                                    java.util.Random rand)

Performs an evaluation of the regressor using the training data set. The evaluation is done by performing cross validation.

Parameters:

folds - the number of folds for cross validation

rand - the source of randomness for generating the cross validation sets

Throws:

UntrainedModelException - if the number of folds given is less than 2

evaluateCrossValidation

public void evaluateCrossValidation(java.util.List<RegressionDataSet> lcds)

Performs an evaluation of the regressor using the training data set, where the folds of the training data set are provided by the user. The folds do not need to be the same sizes, though it is assumed that they are all approximately the same size. It is the caller's responsibility to ensure that the folds are only from the original training data set.

This method exists so that the user can provide very specific folds if they so desire. This can be useful when there is known bias in the data set, such as when caused by duplicate data point values. The caller can then manually make sure duplicate values all occur in the same fold to avoid over-estimating the accuracy of the model.

Parameters:

lcds - the training data set already split into folds

evaluateCrossValidation

public void evaluateCrossValidation(java.util.List<RegressionDataSet> lcds,
                                    java.util.List<RegressionDataSet> trainCombinations)

Note: Most people should never need to call this method. Make sure you understand what you are doing before you do.

Performs an evaluation of the regressor using the training data set, where the folds of the training data set, and their combinations, are provided by the user. The folds do not need to be the same sizes, though it is assumed that they are all approximately the same size - and the the training combination corresponding to each index will be the sum of the folds in the other indices. It is the caller's responsibility to ensure that the folds are only from the original training data set.

This method exists so that the user can provide very specific folds if they so desire, and when the same folds will be used multiple times. Doing so allows the algorithms called to take advantage of any potential caching of results based on the data set and avoid all possible excessive memory movement. (For example, DataSet.getNumericColumns() may get re-used and benefit from its caching)
The same behavior of this method can be obtained by calling evaluateCrossValidation(java.util.List).

Parameters:

lcds - training data set already split into folds

trainCombinations - each index contains the training data sans the data stored in the fold associated with that index

evaluateTestSet

public void evaluateTestSet(RegressionDataSet testSet)

Performs an evaluation of the regressor using the initial data set to train, and testing on the given data set.

Parameters:

testSet - the data set to perform testing on

addScorer

public void addScorer(RegressionScore scorer)

Adds a new score object that will be used as part of the evaluation when calling evaluateCrossValidation(int, java.util.Random) or evaluateTestSet(jsat.regression.RegressionDataSet). The statistics for the given score are reset on every call, and the mean / standard deviation comes from multiple folds in cross validation.

The score statistics can be obtained from getScoreStats(jsat.regression.evaluation.RegressionScore) after one of the evaluation methods have been called.

Parameters:

scorer - the score method to keep track of.

getScoreStats

public OnLineStatistics getScoreStats(RegressionScore score)

Gets the statistics associated with the given score. If the score is not currently in the model evaluation null will be returned. The object passed in does not need to be the exact same object passed to addScorer(jsat.regression.evaluation.RegressionScore), it only needs to be equal to the object.

Parameters:

score - the score type to get the result statistics

Returns:

the result statistics for the given score, or null if the score is not in th evaluation set

prettyPrintRegressionScores

public void prettyPrintRegressionScores()

Prints out the classification information in a convenient format. If no additional scores were added via the addScorer(RegressionScore) method, nothing will be printed.

getMinError

public double getMinError()

Returns the minimum squared error from all runs.

Returns:

the minimum observed squared error

getMaxError

public double getMaxError()

Returns the maximum squared error observed from all runs.

Returns:

the maximum observed squared error

getMeanError

public double getMeanError()

Returns the mean squared error from all runs.

Returns:

the overall mean squared error

getErrorStndDev

public double getErrorStndDev()

Returns the standard deviation of the error from all runs

Returns:

the overall standard deviation of the errors

getTotalTrainingTime

public long getTotalTrainingTime()

Returns the total number of milliseconds spent training the regressor.

Returns:

the total number of milliseconds spent training the regressor.

getTotalClassificationTime

public long getTotalClassificationTime()

Returns the total number of milliseconds spent performing regression on the testing set.

Returns:

the total number of milliseconds spent performing regression on the testing set.

getRegressor

public Regressor getRegressor()

Returns the regressor that was to be evaluated

Returns:

the regressor original given