Catalano.MachineLearning.Exploration

Class Sarsa

java.lang.Object

Catalano.MachineLearning.Exploration.Sarsa

public class Sarsa
extends java.lang.Object

The class provides implementation of Sarse algorithm, known as on-policy Temporal Difference control.

Constructor Summary

Constructors

Constructor and Description
Sarsa(int states, int actions, IExplorationPolicy explorationPolicy, boolean randomize) Initializes a new instance of the Sarsa class.

Method Summary

Modifier and Type	Method and Description
int	GetAction(int state) Get next action from the specified state.
int	getActions() Amount of possible actions.
double	getDiscountFactor() Get Discount Factor
IExplorationPolicy	getExplorationPolicy() Exploration policy.
double	getLearningRate() Get learning rate
int	getStates() Amount of possible states.
void	setDiscountFactor(double discountFactor) Discount factor, [0, 1].
void	setExplorationPolicy(IExplorationPolicy explorationPolicy) Exploration policy.
void	setLearningRate(double learningRate) Learning rate, [0, 1].
void	UpdateState(int previousState, int previousAction, double reward) Update Q-function's value for the previous state-action pair.
void	UpdateState(int previousState, int previousAction, double reward, int nextState, int nextAction) Update Q-function's value for the previous state-action pair.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Sarsa

public Sarsa(int states,
             int actions,
             IExplorationPolicy explorationPolicy,
             boolean randomize)

Initializes a new instance of the Sarsa class.

Parameters:

states - Amount of possible states.

actions - Amount of possible actions.

explorationPolicy - Exploration policy.

randomize - Randomize action estimates or not. The randomize parameter specifies if initial action estimates should be randomized with small values or not. Randomization of action values may be useful, when greedy exploration policies are used. In this case randomization ensures that actions of the same type are not chosen always.

Method Detail

getStates

public int getStates()

Amount of possible states.

Returns:

States

getActions

public int getActions()

Amount of possible actions.

Returns:

Actions

getExplorationPolicy

public IExplorationPolicy getExplorationPolicy()

Exploration policy.

Returns:

Exploration policy.

setExplorationPolicy

public void setExplorationPolicy(IExplorationPolicy explorationPolicy)

Exploration policy. Policy, which is used to select actions.

Parameters:

explorationPolicy - Exploration policy

getLearningRate

public double getLearningRate()

Get learning rate

Returns:

Learning rate

setLearningRate

public void setLearningRate(double learningRate)

Learning rate, [0, 1]. The value determines the amount of updates Q-function receives during learning. The greater the value, the more updates the function receives. The lower the value, the less updates it receives.

Parameters:

learningRate - Learning rate

getDiscountFactor

public double getDiscountFactor()

Get Discount Factor

Returns:

Discount Factor

setDiscountFactor

public void setDiscountFactor(double discountFactor)

Discount factor, [0, 1]. Discount factor for the expected summary reward. The value serves as multiplier for the expected reward. So if the value is set to 1, then the expected summary reward is not discounted. If the value is getting smaller, then smaller amount of the expected reward is used for actions' estimates update.

Parameters:

discountFactor - Discount Factor

GetAction

public int GetAction(int state)

Get next action from the specified state.

Parameters:

state - Current state to get an action for.

Returns:

Returns the action for the state.

UpdateState

public void UpdateState(int previousState,
                        int previousAction,
                        double reward,
                        int nextState,
                        int nextAction)

Update Q-function's value for the previous state-action pair.

Parameters:

previousState - Current state.

previousAction - Action, which lead from previous to the next state.

reward - Reward value, received by taking specified action from previous state.

nextState - Next state.

nextAction - Next action.

UpdateState

public void UpdateState(int previousState,
                        int previousAction,
                        double reward)

Update Q-function's value for the previous state-action pair.

Parameters:

previousState - Current state.

previousAction - Action, which lead from previous to the next state.

reward - Reward value, received by taking specified action from previous state. Updates Q-function's value for the previous state-action pair in the case if the next state is terminal.