jsat.math.optimization.stochastic

Class AdaGrad

java.lang.Object

jsat.math.optimization.stochastic.AdaGrad

All Implemented Interfaces:

java.io.Serializable, GradientUpdater

public class AdaGrad
extends java.lang.Object
implements GradientUpdater

AdaGrad provides an adaptive learning rate for each individual feature

See: Duchi, J., Hazan, E.,&Singer, Y. (2011). Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research, 12, 2121–2159.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
AdaGrad() Creates a new AdaGrad updater
AdaGrad(AdaGrad toCopy) Copy constructor

Method Summary

Modifier and Type	Method and Description
AdaGrad	clone()
void	setup(int d) Sets up this updater to update a weight vector of dimension d by a gradient of the same dimension
void	update(Vec x, Vec grad, double eta) Updates the weight vector x such that x = x-ηf(grad), where f(grad) is some function on the gradient that effectively returns a new vector.
double	update(Vec x, Vec grad, double eta, double bias, double biasGrad) Updates the weight vector x such that x = x-ηf(grad), where f(grad) is some function on the gradient that effectively returns a new vector.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AdaGrad

public AdaGrad()

Creates a new AdaGrad updater

AdaGrad

public AdaGrad(AdaGrad toCopy)

Copy constructor

Parameters:

toCopy - the object to copy

Method Detail

update

public void update(Vec x,
                   Vec grad,
                   double eta)

Description copied from interface: GradientUpdater

Updates the weight vector x such that x = x-ηf(grad), where f(grad) is some function on the gradient that effectively returns a new vector. It is not necessary for the internal implementation to ever explicitly form any of these objects, so long as x is mutated to have the correct result.

Specified by:

update in interface GradientUpdater

Parameters:

x - the vector to mutate such that is has been updated by the gradient

grad - the gradient to update the weight vector x from

eta - the learning rate to apply

update

public double update(Vec x,
                     Vec grad,
                     double eta,
                     double bias,
                     double biasGrad)

Description copied from interface: GradientUpdater

Updates the weight vector x such that x = x-ηf(grad), where f(grad) is some function on the gradient that effectively returns a new vector. It is not necessary for the internal implementation to ever explicitly form any of these objects, so long as x is mutated to have the correct result.

This version of the update method includes two extra parameters to make it easer to use when a scalar bias term is also used

Specified by:

update in interface GradientUpdater

Parameters:

x - the vector to mutate such that is has been updated by the gradient

grad - the gradient to update the weight vector x from

eta - the learning rate to apply

bias - the bias term of the vector

biasGrad - the gradient for the bias term

Returns:

the value to change the bias by, the update being bias = bias - returnValue

clone

public AdaGrad clone()

Specified by:

clone in interface GradientUpdater

Overrides:

clone in class java.lang.Object

setup

public void setup(int d)

Description copied from interface: GradientUpdater

Sets up this updater to update a weight vector of dimension d by a gradient of the same dimension

Specified by:

setup in interface GradientUpdater

Parameters:

d - the dimension of the weight vector that will be updated