jsat.math.optimization.stochastic

Class SGDMomentum

java.lang.Object

jsat.math.optimization.stochastic.SGDMomentum

All Implemented Interfaces:

java.io.Serializable, GradientUpdater

public class SGDMomentum
extends java.lang.Object
implements GradientUpdater

Performs unaltered Stochastic Gradient Decent updates using either standard or Nestrov momentum.

See:

• Bengio, Y., Boulanger-Lewandowski, N.,&Pascanu, R. (2013). Advances in optimizing recurrent networks. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 8624–8628). IEEE. doi:10.1109/ICASSP.2013.6639349
• Sutskever, I., Martens, J., Dahl, G.,&Hinton, G. (2013). On the importance of initialization and momentum in deep learning. JMLR W&CP, 28, 1139–1147.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
SGDMomentum(double momentum) Creates a new SGD with Nestrov Momentum learner
SGDMomentum(double momentum, boolean nestrov) Creates a new SGD with Momentum learner
SGDMomentum(SGDMomentum toCopy) Copy constructor

Method Summary

Modifier and Type	Method and Description
SGDMomentum	clone()
double	getMomentum()
void	setMomentum(double momentum) Sets the momentum for accumulating gradients.
void	setup(int d) Sets up this updater to update a weight vector of dimension d by a gradient of the same dimension
void	update(Vec x, Vec grad, double eta) Updates the weight vector x such that x = x-ηf(grad), where f(grad) is some function on the gradient that effectively returns a new vector.
double	update(Vec x, Vec grad, double eta, double bias, double biasGrad) Updates the weight vector x such that x = x-ηf(grad), where f(grad) is some function on the gradient that effectively returns a new vector.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SGDMomentum

public SGDMomentum(double momentum,
                   boolean nestrov)

Creates a new SGD with Momentum learner

Parameters:

momentum - the amount of momentum to use

nestrov - true to use Nestrov momentum, false for standard.

SGDMomentum

public SGDMomentum(double momentum)

Creates a new SGD with Nestrov Momentum learner

Parameters:

momentum - the amount of momentum to use

SGDMomentum

public SGDMomentum(SGDMomentum toCopy)

Copy constructor

Parameters:

toCopy - the object to copy

Method Detail

setMomentum

public void setMomentum(double momentum)

Sets the momentum for accumulating gradients.

Parameters:

momentum - the momentum buildup term in (0, 1)

getMomentum

public double getMomentum()

Returns:

the momentum buildup term

update

public void update(Vec x,
                   Vec grad,
                   double eta)

Description copied from interface: GradientUpdater

Updates the weight vector x such that x = x-ηf(grad), where f(grad) is some function on the gradient that effectively returns a new vector. It is not necessary for the internal implementation to ever explicitly form any of these objects, so long as x is mutated to have the correct result.

Specified by:

update in interface GradientUpdater

Parameters:

x - the vector to mutate such that is has been updated by the gradient

grad - the gradient to update the weight vector x from

eta - the learning rate to apply

update

public double update(Vec x,
                     Vec grad,
                     double eta,
                     double bias,
                     double biasGrad)

Description copied from interface: GradientUpdater

Updates the weight vector x such that x = x-ηf(grad), where f(grad) is some function on the gradient that effectively returns a new vector. It is not necessary for the internal implementation to ever explicitly form any of these objects, so long as x is mutated to have the correct result.

This version of the update method includes two extra parameters to make it easer to use when a scalar bias term is also used

Specified by:

update in interface GradientUpdater

Parameters:

x - the vector to mutate such that is has been updated by the gradient

grad - the gradient to update the weight vector x from

eta - the learning rate to apply

bias - the bias term of the vector

biasGrad - the gradient for the bias term

Returns:

the value to change the bias by, the update being bias = bias - returnValue

clone

public SGDMomentum clone()

Specified by:

clone in interface GradientUpdater

Overrides:

clone in class java.lang.Object

setup

public void setup(int d)

Description copied from interface: GradientUpdater

Sets up this updater to update a weight vector of dimension d by a gradient of the same dimension

Specified by:

setup in interface GradientUpdater

Parameters:

d - the dimension of the weight vector that will be updated