jsat.math.optimization.stochastic

Class Adam

java.lang.Object

jsat.math.optimization.stochastic.Adam

All Implemented Interfaces:

java.io.Serializable, GradientUpdater

public class Adam
extends java.lang.Object
implements GradientUpdater

Adam is inspired by RMSProp and AdaGrad, where the former can be seen as a special case of Adam. Adam has been shown to work well in training neural networks, and still converges well with sparse gradients.
NOTE: that while it will converge, Adam dose not support sparse updates. So runtime when in highly sparse environments will be hampered.

See: Kingma, D. P.,&Ba, J. L. (2015). Adam: A Method for Stochastic Optimization. In ICLR.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static double	DEFAULT_ALPHA
static double	DEFAULT_BETA_1
static double	DEFAULT_BETA_2
static double	DEFAULT_EPS
static double	DEFAULT_LAMBDA

Constructor Summary

Constructors

Constructor and Description
Adam()
Adam(Adam toCopy) Copy constructor
Adam(double alpha, double beta_1, double beta_2, double eps, double lambda)

Method Summary

Modifier and Type	Method and Description
Adam	clone()
void	setup(int d) Sets up this updater to update a weight vector of dimension d by a gradient of the same dimension
void	update(Vec x, Vec grad, double eta) Updates the weight vector x such that x = x-ηf(grad), where f(grad) is some function on the gradient that effectively returns a new vector.
double	update(Vec x, Vec grad, double eta, double bias, double biasGrad) Updates the weight vector x such that x = x-ηf(grad), where f(grad) is some function on the gradient that effectively returns a new vector.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_ALPHA

public static final double DEFAULT_ALPHA

See Also:

Constant Field Values

DEFAULT_BETA_1

public static final double DEFAULT_BETA_1

See Also:

Constant Field Values

DEFAULT_BETA_2

public static final double DEFAULT_BETA_2

See Also:

Constant Field Values

DEFAULT_EPS

public static final double DEFAULT_EPS

See Also:

Constant Field Values

DEFAULT_LAMBDA

public static final double DEFAULT_LAMBDA

See Also:

Constant Field Values

Constructor Detail

Adam

public Adam()

Adam

public Adam(double alpha,
            double beta_1,
            double beta_2,
            double eps,
            double lambda)

Adam

public Adam(Adam toCopy)

Copy constructor

Parameters:

toCopy - the object to copy

Method Detail

update

public void update(Vec x,
                   Vec grad,
                   double eta)

Description copied from interface: GradientUpdater

Updates the weight vector x such that x = x-ηf(grad), where f(grad) is some function on the gradient that effectively returns a new vector. It is not necessary for the internal implementation to ever explicitly form any of these objects, so long as x is mutated to have the correct result.

Specified by:

update in interface GradientUpdater

Parameters:

x - the vector to mutate such that is has been updated by the gradient

grad - the gradient to update the weight vector x from

eta - the learning rate to apply

update

public double update(Vec x,
                     Vec grad,
                     double eta,
                     double bias,
                     double biasGrad)

Description copied from interface: GradientUpdater

Updates the weight vector x such that x = x-ηf(grad), where f(grad) is some function on the gradient that effectively returns a new vector. It is not necessary for the internal implementation to ever explicitly form any of these objects, so long as x is mutated to have the correct result.

This version of the update method includes two extra parameters to make it easer to use when a scalar bias term is also used

Specified by:

update in interface GradientUpdater

Parameters:

x - the vector to mutate such that is has been updated by the gradient

grad - the gradient to update the weight vector x from

eta - the learning rate to apply

bias - the bias term of the vector

biasGrad - the gradient for the bias term

Returns:

the value to change the bias by, the update being bias = bias - returnValue

clone

public Adam clone()

Specified by:

clone in interface GradientUpdater

Overrides:

clone in class java.lang.Object

setup

public void setup(int d)

Description copied from interface: GradientUpdater

Sets up this updater to update a weight vector of dimension d by a gradient of the same dimension

Specified by:

setup in interface GradientUpdater

Parameters:

d - the dimension of the weight vector that will be updated