jsat.classifiers.linear

Class StochasticMultinomialLogisticRegression

java.lang.Object

jsat.classifiers.linear.StochasticMultinomialLogisticRegression

All Implemented Interfaces:

java.io.Serializable, java.lang.Cloneable, Classifier, Parameterized, SimpleWeightVectorModel

public class StochasticMultinomialLogisticRegression
extends java.lang.Object
implements Classifier, Parameterized, SimpleWeightVectorModel

This is a Stochastic implementation of Multinomial Logistic Regression. It supports regularization from several different priors, and performs prior updates in a lazy fashion to avoid destroying the sparsity of training inputs.
Algorithm is based on the technical report:
Carpenter, B. (2008). Lazy Sparse Stochastic Gradient Descent for Regularized Mutlinomial Logistic Regression. Retrieved from http://lingpipe-blog.com/lingpipe-white-papers/

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	StochasticMultinomialLogisticRegression.Prior Represents a prior of the coefficients that can be applied to perform regularization.

Constructor Summary

Constructors

Constructor and Description
StochasticMultinomialLogisticRegression() Creates a new Stochastic Multinomial Logistic Regression that uses a StochasticMultinomialLogisticRegression.Prior.GAUSSIAN prior with a regularization scale of 1e-6.
StochasticMultinomialLogisticRegression(double initialLearningRate, int epochs) Creates a new Stochastic Multinomial Logistic Regression that uses a StochasticMultinomialLogisticRegression.Prior.GAUSSIAN prior with a regularization scale of 1e-6.
StochasticMultinomialLogisticRegression(double initialLearningRate, int epochs, double regularization, StochasticMultinomialLogisticRegression.Prior prior) Creates a new Stochastic Multinomial Logistic Regression object

Method Summary

Modifier and Type	Method and Description
CategoricalResults	classify(DataPoint data) Performs classification on the given data point.
Classifier	clone()
double	getAlpha() Returns the extra parameter value
double	getBias(int index) Returns the bias term used with the weight vector for the given class index.
Vec	getCoefficientVector(int id) Returns the raw coefficient vector used without the bias term.
int	getEpochs() Returns the maximum number of epochs
double	getInitialLearningRate() Returns the current initial learning rate
DecayRate	getLearningRateDecay() Returns the decay rate in use
int	getMiniBatchSize() Returns the number of data points used to perform each gradient update
StochasticMultinomialLogisticRegression.Prior	getPrior() Returns the prior used for regularization
Vec	getRawWeight(int index) Returns the raw weight vector associated with the given class index.
double	getRegularization() Returns the regularization coefficient in use
double	getTolerance() Returns the minimum tolerance for early stopping.
boolean	isClipping() Returns whether or not coefficient clipping is on.
boolean	isStandardized() Returns whether or not the input is standardized for the priors
boolean	isUseBias() Returns true if the bias term is in use
int	numWeightsVecs() Returns the number of weight vectors that can be returned.
void	setAlpha(double alpha) Sets the extra parameter alpha.
void	setClipping(boolean clipping) Sets whether or not the clip changes in coefficient values caused by regularization so that they can not make the coefficients go from positive to negative or negative to positive.
void	setEpochs(int epochs) Sets the maximum number of epochs that occur in each iteration.
void	setInitialLearningRate(double initialLearningRate) Sets the initial learning rate to use for the first epoch.
void	setLearningRateDecay(DecayRate learningRateDecay) Sets the decay rate used to reduce the learning rate after each epoch.
void	setMiniBatchSize(int miniBatchSize) Sets the amount of data points used to form each gradient update.
void	setPrior(StochasticMultinomialLogisticRegression.Prior prior) Sets the prior used to perform regularization
void	setRegularization(double regularization) Sets the coefficient applied to the regularization penalty at each update.
void	setStandardized(boolean standardized) Sets whether or not to perform implicit standardization of the feature values when performing regularization by the prior.
void	setTolerance(double tolerance) Sets the tolerance that determines when the training stops early because the change has become too insignificant.
void	setUseBias(boolean useBias) Sets whether or not to learn the bias term for a model.
boolean	supportsWeightedData() Indicates whether the model knows how to train using weighted data points.
void	train(ClassificationDataSet dataSet) Trains the classifier and constructs a model for classification using the given data set.
void	train(ClassificationDataSet dataSet, boolean parallel) Trains the classifier and constructs a model for classification using the given data set.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface jsat.parameters.Parameterized

getParameter, getParameters

Constructor Detail

StochasticMultinomialLogisticRegression

public StochasticMultinomialLogisticRegression(double initialLearningRate,
                                               int epochs,
                                               double regularization,
                                               StochasticMultinomialLogisticRegression.Prior prior)

Creates a new Stochastic Multinomial Logistic Regression object

Parameters:

initialLearningRate - the initial learning rate to use

epochs - the maximum number of training epochs to go through

regularization - the scale factor applied to the regularization term

prior - the prior to use for regularization

StochasticMultinomialLogisticRegression

public StochasticMultinomialLogisticRegression(double initialLearningRate,
                                               int epochs)

Creates a new Stochastic Multinomial Logistic Regression that uses a StochasticMultinomialLogisticRegression.Prior.GAUSSIAN prior with a regularization scale of 1e-6.

Parameters:

initialLearningRate - the initial learning rate to use

epochs - the maximum number of training epochs to go through

StochasticMultinomialLogisticRegression

public StochasticMultinomialLogisticRegression()

Creates a new Stochastic Multinomial Logistic Regression that uses a StochasticMultinomialLogisticRegression.Prior.GAUSSIAN prior with a regularization scale of 1e-6. It will do at most 50 epochs with a learning rate of 0.1

Method Detail

setUseBias

public void setUseBias(boolean useBias)

Sets whether or not to learn the bias term for a model. If no bias term is in use, the model learned must pass through the origin of the world. The use of the bias term is very important for low dimensional problems, but less so for many higher dimensional problems.

Parameters:

useBias - true if the bias term should be used, false otherwise

isUseBias

public boolean isUseBias()

Returns true if the bias term is in use

Returns:

true if the bias term is in use

setEpochs

public void setEpochs(int epochs)

Sets the maximum number of epochs that occur in each iteration. Each epoch goes through the whole data set once.

Parameters:

epochs - the maximum number of epochs to train

getEpochs

public int getEpochs()

Returns the maximum number of epochs

Returns:

the maximum number of epochs

setAlpha

public void setAlpha(double alpha)

Sets the extra parameter alpha. This is used for some priors that take an extra parameter. This is StochasticMultinomialLogisticRegression.Prior.CAUCHY and StochasticMultinomialLogisticRegression.Prior.ELASTIC. If these two priors are not in use, the value is ignored.

Parameters:

alpha - the extra parameter value to use. Must be positive

getAlpha

public double getAlpha()

Returns the extra parameter value

Returns:

the extra parameter value

setClipping

public void setClipping(boolean clipping)

Sets whether or not the clip changes in coefficient values caused by regularization so that they can not make the coefficients go from positive to negative or negative to positive. If clipping is on, the value will go to zero instead. If off, the value will be allowed to change signs.
If there is no regularization, this has no impact.

Parameters:

clipping - true if clipping should be used, false otherwise

isClipping

public boolean isClipping()

Returns whether or not coefficient clipping is on.

Returns:

true if clipping is on.

setInitialLearningRate

public void setInitialLearningRate(double initialLearningRate)

Sets the initial learning rate to use for the first epoch. The learning rate will decay according to the decay rate in use.

Parameters:

initialLearningRate - the initial learning rate to use

getInitialLearningRate

public double getInitialLearningRate()

Returns the current initial learning rate

Returns:

the learning rate in use

setLearningRateDecay

public void setLearningRateDecay(DecayRate learningRateDecay)

Sets the decay rate used to reduce the learning rate after each epoch.

Parameters:

learningRateDecay - the decay rate to use

getLearningRateDecay

public DecayRate getLearningRateDecay()

Returns the decay rate in use

Returns:

the decay rate in use

setRegularization

public void setRegularization(double regularization)

Sets the coefficient applied to the regularization penalty at each update. This is usual set to a small value less than 1. If set to zero, it effectively turns off the use of regularization.

Parameters:

regularization - the non negative regularization coefficient to apply

getRegularization

public double getRegularization()

Returns the regularization coefficient in use

Returns:

the regularization coefficient in use

setPrior

public void setPrior(StochasticMultinomialLogisticRegression.Prior prior)

Sets the prior used to perform regularization

Parameters:

prior - the prior to use

getPrior

public StochasticMultinomialLogisticRegression.Prior getPrior()

Returns the prior used for regularization

Returns:

the prior used

setTolerance

public void setTolerance(double tolerance)

Sets the tolerance that determines when the training stops early because the change has become too insignificant.

Parameters:

tolerance - the minimum change in log likelihood to stop training

getTolerance

public double getTolerance()

Returns the minimum tolerance for early stopping.

Returns:

the minimum change in log likelihood to stop training

setStandardized

public void setStandardized(boolean standardized)

Sets whether or not to perform implicit standardization of the feature values when performing regularization by the prior. If set on, the input data will be adjusted to have zero mean and unit variance. This is done without destroying sparsity. If there is not regularization, this parameter has no impact.

Parameters:

standardized - true if the input will be standardized, false if ti will be left as is.

isStandardized

public boolean isStandardized()

Returns whether or not the input is standardized for the priors

Returns:

true if the input is standardized for the priors

setMiniBatchSize

public void setMiniBatchSize(int miniBatchSize)

Sets the amount of data points used to form each gradient update. Increasing the batch size can help convergence. By default, a mini batch size of 1 is used.

Parameters:

miniBatchSize - the number of data points used to perform each update

getMiniBatchSize

public int getMiniBatchSize()

Returns the number of data points used to perform each gradient update

Returns:

the number of data points used to perform each gradient update

getRawWeight

public Vec getRawWeight(int index)

Description copied from interface: SimpleWeightVectorModel

Returns the raw weight vector associated with the given class index. If the given class is an implicit zero vector, a ConstantVector object may be returned.
Do not alter the returned weight vector, as it will change the model's values.

If a regression problem, only index = 0 should be used

Specified by:

getRawWeight in interface SimpleWeightVectorModel

Parameters:

index - the class index to get the weight vector for

Returns:

the weight vector used for the specified class

getBias

public double getBias(int index)

Description copied from interface: SimpleWeightVectorModel

Returns the bias term used with the weight vector for the given class index. If the model does not support or was not trained with bias weights, 0 will be returned.

If a regression problem, only index = 0 should be used

Specified by:

getBias in interface SimpleWeightVectorModel

Parameters:

index - the class index to get the weight vector for

Returns:

the bias term for the specified class

numWeightsVecs

public int numWeightsVecs()

Description copied from interface: SimpleWeightVectorModel

Returns the number of weight vectors that can be returned. For binary classification problems the value may be 1 if only a single weight vector's sign is used to determine the class. For multi-class problems, the weight vector count includes the implicit zero vector (if one is being used).

Specified by:

numWeightsVecs in interface SimpleWeightVectorModel

Returns:

the number of weight vectors for which SimpleWeightVectorModel.getRawWeight(int) can be called.

classify

public CategoricalResults classify(DataPoint data)

Description copied from interface: Classifier

Performs classification on the given data point.

Specified by:

classify in interface Classifier

Parameters:

data - the data point to classify

Returns:

the results of the classification.

train

public void train(ClassificationDataSet dataSet,
                  boolean parallel)

Description copied from interface: Classifier

Trains the classifier and constructs a model for classification using the given data set. If the training method knows how, it will used the threadPool to conduct training in parallel. This method will block until the training has completed.

Specified by:

train in interface Classifier

Parameters:

dataSet - the data set to train on

parallel - true if multiple threads should be used to train the model. false if it should be done in a single threaded manner.

train

public void train(ClassificationDataSet dataSet)

Description copied from interface: Classifier

Trains the classifier and constructs a model for classification using the given data set.

Specified by:

train in interface Classifier

Parameters:

dataSet - the data set to train on

supportsWeightedData

public boolean supportsWeightedData()

Description copied from interface: Classifier

Indicates whether the model knows how to train using weighted data points. If it does, the model will train assuming the weights. The values returned by this method may change depending on the parameters set for the model.

Specified by:

supportsWeightedData in interface Classifier

Returns:

true if the model supports weighted data, false otherwise

getCoefficientVector

public Vec getCoefficientVector(int id)

Returns the raw coefficient vector used without the bias term. For a multinomial Logistic model, there are C-1 coefficient vectors. C is the number of output classes. Altering the returned vector will alter the model. The i'th index of the vector corresponds to the weight therm for the i'th index in an input.

Parameters:

id - which coefficient vector to obtain

Returns:

the vector of variable coefficients.

clone

public Classifier clone()

Specified by:

clone in interface Classifier

Overrides:

clone in class java.lang.Object