Catalano.MachineLearning.Classification.DecisionTrees.Learning

Class RandomForest

java.lang.Object

Catalano.MachineLearning.Classification.DecisionTrees.Learning.RandomForest

All Implemented Interfaces:

IClassifier, java.io.Serializable, java.lang.Cloneable

public class RandomForest
extends java.lang.Object
implements IClassifier, java.io.Serializable

Random forest for classification. Random forest is an ensemble classifier that consists of many decision trees and outputs the majority vote of individual trees. The method combines bagging idea and the random selection of features.

Each tree is constructed using the following algorithm:

1. If the number of cases in the training set is N, randomly sample N cases with replacement from the original data. This sample will be the training set for growing the tree.
2. If there are M input variables, a number m << M is specified such that at each node, m variables are selected at random out of the M and the best split on these m is used to split the node. The value of m is held constant during the forest growing.
3. Each tree is grown to the largest extent possible. There is no pruning.

The advantages of random forest are:

• For many data sets, it produces a highly accurate classifier.
• It runs efficiently on large data sets.
• It can handle thousands of input variables without variable deletion.
• It gives estimates of what variables are important in the classification.
• It generates an internal unbiased estimate of the generalization error as the forest building progresses.
• It has an effective method for estimating missing data and maintains accuracy when a large proportion of the data are missing.

The disadvantages are

• Random forests are prone to over-fitting for some datasets. This is even more pronounced on noisy data.
• For data including categorical variables with different number of levels, random forests are biased in favor of those attributes with more levels. Therefore, the variable importance scores from random forest are not reliable for this type of data.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	RandomForest.RandomSelection Method for choose number of random feature.

Constructor Summary

Constructors

Constructor and Description
RandomForest() Initializes a new instance of the RandomForest class.
RandomForest(DecisionVariable[] attributes) Initializes a new instance of the RandomForest class.
RandomForest(DecisionVariable[] attributes, int T) Initializes a new instance of the RandomForest class.
RandomForest(DecisionVariable[] attributes, int T, int M) Initializes a new instance of the RandomForest class.
RandomForest(DecisionVariable[] attributes, int T, int M, DecisionTree.SplitRule rule) Initializes a new instance of the RandomForest class.
RandomForest(DecisionVariable[] attributes, int T, RandomForest.RandomSelection randomSelection) Initializes a new instance of the RandomForest class.
RandomForest(DecisionVariable[] attributes, int T, RandomForest.RandomSelection randomSelection, DecisionTree.SplitRule rule) Initializes a new instance of the RandomForest class.
RandomForest(int T) Initializes a new instance of the RandomForest class.
RandomForest(int T, int M) Initializes a new instance of the RandomForest class.
RandomForest(int T, int M, DecisionTree.SplitRule rule) Initializes a new instance of the RandomForest class.
RandomForest(int T, RandomForest.RandomSelection randomSelection) Initializes a new instance of the RandomForest class.
RandomForest(int T, RandomForest.RandomSelection randomSelection, DecisionTree.SplitRule rule) Initializes a new instance of the RandomForest class.

Method Summary

Modifier and Type	Method and Description
IClassifier	clone() Clone of the object.
double	error() Returns the out-of-bag estimation of error rate.
DecisionTree.SplitRule	getRule() Get split rule.
double[]	importance() Returns the variable importance.
void	Learn(DatasetClassification dataset) Learn from a dataset.
void	Learn(double[][] input, int[] output) Learn.
int	Predict(double[] feature) Predict.
int	Predict(double[] feature, double[] posteriori)
void	setRule(DecisionTree.SplitRule rule) Set split rule.
int	size() Returns the number of trees in the model.
void	trim(int T) Trims the tree model set to a smaller size in case of over-fitting.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RandomForest

public RandomForest()

Initializes a new instance of the RandomForest class.

RandomForest

public RandomForest(int T)

Initializes a new instance of the RandomForest class.

Parameters:

T - the number of trees.

RandomForest

public RandomForest(int T,
                    int M)

Initializes a new instance of the RandomForest class.

Parameters:

T - the number of trees.

M - the number of random selected features to be used to determine the decision at a node of the tree. floor(sqrt(dim)) seems to give generally good performance, where dim is the number of variables.

RandomForest

public RandomForest(int T,
                    int M,
                    DecisionTree.SplitRule rule)

Initializes a new instance of the RandomForest class.

Parameters:

T - the number of trees.

M - the number of random selected features to be used to determine the decision at a node of the tree. floor(sqrt(dim)) seems to give generally good performance, where dim is the number of variables.

rule - Split rule decision tree.

RandomForest

public RandomForest(int T,
                    RandomForest.RandomSelection randomSelection)

Initializes a new instance of the RandomForest class.

Parameters:

T - the number of trees.

randomSelection - the number of random selected features.

RandomForest

public RandomForest(int T,
                    RandomForest.RandomSelection randomSelection,
                    DecisionTree.SplitRule rule)

Initializes a new instance of the RandomForest class.

Parameters:

T - the number of trees.

randomSelection - the number of random selected features.

RandomForest

public RandomForest(DecisionVariable[] attributes)

Initializes a new instance of the RandomForest class.

Parameters:

attributes - the attribute properties.

RandomForest

public RandomForest(DecisionVariable[] attributes,
                    int T)

Initializes a new instance of the RandomForest class.

Parameters:

attributes - the attribute properties.

T - the number of trees.

RandomForest

public RandomForest(DecisionVariable[] attributes,
                    int T,
                    int M)

Initializes a new instance of the RandomForest class.

Parameters:

attributes - the attribute properties.

T - the number of trees.

M - the number of random selected features to be used to determine the decision at a node of the tree. floor(sqrt(dim)) seems to give generally good performance, where dim is the number of variables.

RandomForest

public RandomForest(DecisionVariable[] attributes,
                    int T,
                    int M,
                    DecisionTree.SplitRule rule)

Initializes a new instance of the RandomForest class.

Parameters:

attributes - the attribute properties.

T - the number of trees.

M - the number of random selected features.

rule - Split rule decision tree.

RandomForest

public RandomForest(DecisionVariable[] attributes,
                    int T,
                    RandomForest.RandomSelection randomSelection)

Initializes a new instance of the RandomForest class.

Parameters:

attributes - the attribute properties.

T - the number of trees.

randomSelection - The method for create random selected features to be used to determine the decision at a node of the tree. floor(sqrt(dim)) seems to give generally good performance, where dim is the number of variables.

RandomForest

public RandomForest(DecisionVariable[] attributes,
                    int T,
                    RandomForest.RandomSelection randomSelection,
                    DecisionTree.SplitRule rule)

Initializes a new instance of the RandomForest class.

Parameters:

attributes - the attribute properties.

T - the number of trees.

randomSelection - The method for create random selected features to be used to determine the decision at a node of the tree. floor(sqrt(dim)) seems to give generally good performance, where dim is the number of variables.

rule - SplitRule decision tree.

Method Detail

error

public double error()

Returns the out-of-bag estimation of error rate. The OOB estimate is quite accurate given that enough trees have been grown. Otherwise the OOB estimate can bias upward.

Returns:

the out-of-bag estimation of error rate

importance

public double[] importance()

Returns the variable importance. Every time a split of a node is made on variable the (GINI, information gain, etc.) impurity criterion for the two descendent nodes is less than the parent node. Adding up the decreases for each individual variable over all trees in the forest gives a fast measure of variable importance that is often very consistent with the permutation importance measure.

Returns:

the variable importance

size

public int size()

Returns the number of trees in the model.

Returns:

the number of trees in the model

getRule

public DecisionTree.SplitRule getRule()

Get split rule.

Returns:

Split rule.

setRule

public void setRule(DecisionTree.SplitRule rule)

Set split rule.

Parameters:

rule - Split rule.

trim

public void trim(int T)

Trims the tree model set to a smaller size in case of over-fitting. Or if extra decision trees in the model don't improve the performance, we may remove them to reduce the model size and also improve the speed of prediction.

Parameters:

T - the new (smaller) size of tree model set.

Learn

public void Learn(DatasetClassification dataset)

Description copied from interface: IClassifier

Learn from a dataset.

Specified by:

Learn in interface IClassifier

Parameters:

dataset - Dataset.

Learn

public void Learn(double[][] input,
                  int[] output)

Description copied from interface: IClassifier

Learn.

Specified by:

Learn in interface IClassifier

Parameters:

input - Matrix of features.

output - Labels.

Predict

public int Predict(double[] feature)

Description copied from interface: IClassifier

Predict.

Specified by:

Predict in interface IClassifier

Parameters:

feature - Feature.

Returns:

Label.

Predict

public int Predict(double[] feature,
                   double[] posteriori)

clone

public IClassifier clone()

Description copied from interface: IClassifier

Clone of the object.

Specified by:

clone in interface IClassifier

Overrides:

clone in class java.lang.Object

Returns:

A new copy of the object.