jsat.regression

Class StochasticGradientBoosting

java.lang.Object

jsat.regression.StochasticGradientBoosting

All Implemented Interfaces:

java.io.Serializable, java.lang.Cloneable, Parameterized, Regressor

public class StochasticGradientBoosting
extends java.lang.Object
implements Regressor, Parameterized

An implementation of Stochastic Gradient Boosting (SGB) for the Squared Error loss. SGB is also known as Gradient Boosting Machine. There is a specialized version of SGB known as TreeBoost, that is not implemented by this method. SGB is a boosting method derived for regression. It uses many weak learners by attempting to estimate the residual error of all previous learners. It can also use an initial strong learner and use the weak learners to refine the initial estimate.

See papers:
Friedman, J. H. (2002). Stochastic gradient boosting. Computational Statistics&Data Analysis, 38(4), 367–378.

Mohan, A., Chen, Z.,&Weinberger, K. (2011). Web-search ranking with initialized gradient boosted regression trees. Journal of Machine Learning Research, 14,

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static double	DEFAULT_LEARNING_RATE The default value for the setLearningRate(double) is 0.1
static double	DEFAULT_TRAINING_PROPORTION The default value for the training proportion is 0.5.

Constructor Summary

Constructors

Constructor and Description
StochasticGradientBoosting(Regressor weakLearner, int maxIterations) Creates a new SGB learner that is initialized using the weak learner.
StochasticGradientBoosting(Regressor weakLearner, int maxIterations, double learningRate) Creates a new SGB learner that is initialized using the weak learner.
StochasticGradientBoosting(Regressor weakLearner, int maxIterations, double learningRate, double trainingPortion) Creates a new SGB learner that is initialized using the weak learner.
StochasticGradientBoosting(Regressor strongLearner, Regressor weakLearner, int maxIterations, double learningRate, double trainingPortion) Creates a new initialized SGB learner.

Method Summary

Modifier and Type	Method and Description
StochasticGradientBoosting	clone()
double	getLearningRate() Returns the learning rate of the algorithm used to control overfitting.
int	getMaxIterations() Returns the maximum number of iterations used in SGB
double	getTrainingProportion() Returns the fraction of the data points used during each iteration of the training algorithm.
double	regress(DataPoint data)
void	setLearningRate(double learningRate) Sets the learning rate of the algorithm.
void	setMaxIterations(int maxIterations) Sets the maximum number of iterations used in SGB.
void	setTrainingProportion(double trainingProportion) The GB version uses the whole data set at each iteration.
boolean	supportsWeightedData()
void	train(RegressionDataSet dataSet, boolean parallel)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface jsat.regression.Regressor

train

Methods inherited from interface jsat.parameters.Parameterized

getParameter, getParameters

Field Detail

DEFAULT_TRAINING_PROPORTION

public static final double DEFAULT_TRAINING_PROPORTION

The default value for the training proportion is 0.5.

See Also:

Constant Field Values

DEFAULT_LEARNING_RATE

public static final double DEFAULT_LEARNING_RATE

The default value for the setLearningRate(double) is 0.1

See Also:

Constant Field Values

Constructor Detail

StochasticGradientBoosting

public StochasticGradientBoosting(Regressor strongLearner,
                                  Regressor weakLearner,
                                  int maxIterations,
                                  double learningRate,
                                  double trainingPortion)

Creates a new initialized SGB learner.

Parameters:

strongLearner - the powerful learner to refine with weak learners

weakLearner - the weak learner to fit to the residuals in each iteration

maxIterations - the maximum number of algorithm iterations to perform

learningRate - the multiplier to apply to the weak learners

trainingPortion - the proportion of the data set to use for each iteration of learning

StochasticGradientBoosting

public StochasticGradientBoosting(Regressor weakLearner,
                                  int maxIterations,
                                  double learningRate,
                                  double trainingPortion)

Creates a new SGB learner that is initialized using the weak learner.

Parameters:

weakLearner - the weak learner to fit to the residuals in each iteration

maxIterations - the maximum number of algorithm iterations to perform

learningRate - the multiplier to apply to the weak learners

trainingPortion - the proportion of the data set to use for each iteration of learning

StochasticGradientBoosting

public StochasticGradientBoosting(Regressor weakLearner,
                                  int maxIterations,
                                  double learningRate)

Creates a new SGB learner that is initialized using the weak learner.

Parameters:

weakLearner - the weak learner to fit to the residuals in each iteration

maxIterations - the maximum number of algorithm iterations to perform

learningRate - the multiplier to apply to the weak learners

StochasticGradientBoosting

public StochasticGradientBoosting(Regressor weakLearner,
                                  int maxIterations)

Creates a new SGB learner that is initialized using the weak learner.

Parameters:

weakLearner - the weak learner to fit to the residuals in each iteration

maxIterations - the maximum number of algorithm iterations to perform

Method Detail

setMaxIterations

public void setMaxIterations(int maxIterations)

Sets the maximum number of iterations used in SGB.

Parameters:

maxIterations - the maximum number of algorithm iterations to perform

getMaxIterations

public int getMaxIterations()

Returns the maximum number of iterations used in SGB

Returns:

the maximum number of algorithm iterations to perform

setLearningRate

public void setLearningRate(double learningRate)

Sets the learning rate of the algorithm. The GB version uses a learning rate of 1. SGB uses a learning rate in (0,1) to avoid overfitting. The learning rate is multiplied by the output of each weak learner to reduce its contribution.

Parameters:

learningRate - the multiplier to apply to the weak learners

Throws:

java.lang.ArithmeticException - if the learning rate is not in the range (0, 1]

getLearningRate

public double getLearningRate()

Returns the learning rate of the algorithm used to control overfitting.

Returns:

the learning rate multiplier applied to the weak learner outputs

setTrainingProportion

public void setTrainingProportion(double trainingProportion)

The GB version uses the whole data set at each iteration. SGB can use a fraction of the data set at each iteration in order to reduce overfitting and add randomness.

Parameters:

trainingProportion - the fraction of training the data set to use for each iteration of SGB

Throws:

java.lang.ArithmeticException - if the trainingPortion is not a valid fraction in (0, 1]

getTrainingProportion

public double getTrainingProportion()

Returns the fraction of the data points used during each iteration of the training algorithm.

Returns:

the fraction of the training data set to use for each iteration of SGB

regress

public double regress(DataPoint data)

Specified by:

regress in interface Regressor

train

public void train(RegressionDataSet dataSet,
                  boolean parallel)

Specified by:

train in interface Regressor

supportsWeightedData

public boolean supportsWeightedData()

Specified by:

supportsWeightedData in interface Regressor

clone

public StochasticGradientBoosting clone()

Specified by:

clone in interface Regressor

Overrides:

clone in class java.lang.Object