org.encog.neural.networks.training.propagation.back

Class Backpropagation

java.lang.Object

org.encog.ml.train.BasicTraining

org.encog.neural.networks.training.propagation.Propagation

org.encog.neural.networks.training.propagation.back.Backpropagation

All Implemented Interfaces:

MLTrain, BatchSize, LearningRate, Momentum, GradientWorkerOwner, Train, MultiThreadable

public class Backpropagation
extends Propagation
implements Momentum, LearningRate

This class implements a backpropagation training algorithm for feed forward neural networks. It is used in the same manner as any other training class that implements the Train interface. Backpropagation is a common neural network training algorithm. It works by analyzing the error of the output of the neural network. Each neuron in the output layer's contribution, according to weight, to this error is determined. These weights are then adjusted to minimize this error. This process continues working its way backwards through the layers of the neural network. This implementation of the backpropagation algorithm uses both momentum and a learning rate. The learning rate specifies the degree to which the weight matrixes will be modified through each iteration. The momentum specifies how much the previous learning iteration affects the current. To use no momentum at all specify zero. One primary problem with backpropagation is that the magnitude of the partial derivative is often detrimental to the training of the neural network. The other propagation methods of Manhatten and Resilient address this issue in different ways. In general, it is suggested that you use the resilient propagation technique for most Encog training tasks over back propagation.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	LAST_DELTA The resume key for backpropagation.

Constructor Summary

Constructors

Constructor and Description
Backpropagation(ContainsFlat network, MLDataSet training) Create a class to train using backpropagation.
Backpropagation(ContainsFlat network, MLDataSet training, double theLearnRate, double theMomentum)

Method Summary

Modifier and Type	Method and Description
boolean	canContinue()
double[]	getLastDelta()
double	getLearningRate()
double	getMomentum()
void	initOthers() Perform training method specific init.
boolean	isNesterovUpdate()
boolean	isValidResume(TrainingContinuation state) Determine if the specified continuation object is valid to resume with.
TrainingContinuation	pause() Pause the training.
void	resume(TrainingContinuation state) Resume training.
void	setLearningRate(double rate) Set the learning rate, this is value is essentially a percent.
void	setMomentum(double m) Set the momentum for training.
void	setNesterovUpdate(boolean nesterovUpdate)
double	updateWeight(double[] gradients, double[] lastGradient, int index) Update a weight.
double	updateWeight(double[] gradients, double[] lastGradient, int index, double dropoutRate) Update a weight.

Methods inherited from class org.encog.neural.networks.training.propagation.Propagation

calculateGradients, finishTraining, finishTraining, fixFlatSpot, getBatchSize, getCurrentFlatNetwork, getDropoutRate, getL1, getL2, getLastGradient, getMethod, getThreadCount, iteration, iteration, report, rollIteration, setBatchSize, setDroupoutRate, setErrorFunction, setL1, setL2, setThreadCount

Methods inherited from class org.encog.ml.train.BasicTraining

addStrategy, getError, getImplementationType, getIteration, getStrategies, getTraining, isTrainingDone, postIteration, preIteration, setError, setIteration, setTraining

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.encog.ml.train.MLTrain

addStrategy, getError, getImplementationType, getIteration, getStrategies, getTraining, isTrainingDone, setError, setIteration

Field Detail

LAST_DELTA

public static final java.lang.String LAST_DELTA

The resume key for backpropagation.

See Also:

Constant Field Values

Constructor Detail

Backpropagation

public Backpropagation(ContainsFlat network,
                       MLDataSet training)

Create a class to train using backpropagation. Use auto learn rate and momentum. Use the CPU to train.

Parameters:

network - The network that is to be trained.

training - The training data to be used for backpropagation.

Backpropagation

public Backpropagation(ContainsFlat network,
                       MLDataSet training,
                       double theLearnRate,
                       double theMomentum)

Parameters:

network - The network that is to be trained

training - The training set

theLearnRate - The rate at which the weight matrix will be adjusted based on learning.

theMomentum - The influence that previous iteration's training deltas will have on the current iteration.

Method Detail

canContinue

public boolean canContinue()

Specified by:

canContinue in interface MLTrain

Returns:

True if the training can be paused, and later continued.

getLastDelta

public double[] getLastDelta()

Returns:

The last delta values.

getLearningRate

public double getLearningRate()

Specified by:

getLearningRate in interface LearningRate

Returns:

The learning rate, this is value is essentially a percent. It is the degree to which the gradients are applied to the weight matrix to allow learning.

getMomentum

public double getMomentum()

Specified by:

getMomentum in interface Momentum

Returns:

The momentum for training. This is the degree to which changes from which the previous training iteration will affect this training iteration. This can be useful to overcome local minima.

isValidResume

public boolean isValidResume(TrainingContinuation state)

Determine if the specified continuation object is valid to resume with.

Parameters:

state - The continuation object to check.

Returns:

True if the specified continuation object is valid for this training method and network.

pause

public TrainingContinuation pause()

Pause the training.

Specified by:

pause in interface MLTrain

Returns:

A training continuation object to continue with.

resume

public void resume(TrainingContinuation state)

Resume training.

Specified by:

resume in interface MLTrain

Parameters:

state - The training state to return to.

setLearningRate

public void setLearningRate(double rate)

Set the learning rate, this is value is essentially a percent. It is the degree to which the gradients are applied to the weight matrix to allow learning.

Specified by:

setLearningRate in interface LearningRate

Parameters:

rate - The learning rate.

setMomentum

public void setMomentum(double m)

Set the momentum for training. This is the degree to which changes from which the previous training iteration will affect this training iteration. This can be useful to overcome local minima.

Specified by:

setMomentum in interface Momentum

Parameters:

m - The momentum.

updateWeight

public double updateWeight(double[] gradients,
                           double[] lastGradient,
                           int index)

Update a weight.

Specified by:

updateWeight in class Propagation

Parameters:

gradients - The gradients.

lastGradient - The last gradients.

index - The index.

Returns:

The weight delta.

updateWeight

public double updateWeight(double[] gradients,
                           double[] lastGradient,
                           int index,
                           double dropoutRate)

Update a weight.

Specified by:

updateWeight in class Propagation

Parameters:

gradients - The gradients.

lastGradient - The last gradients.

index - The index.

dropoutRate - The dropout rate.

Returns:

The weight delta.

initOthers

public void initOthers()

Perform training method specific init.

Specified by:

initOthers in class Propagation

isNesterovUpdate

public boolean isNesterovUpdate()

setNesterovUpdate

public void setNesterovUpdate(boolean nesterovUpdate)