edu.uci.jforests.eval.ranking

Class TRiskAwareSAROEval

java.lang.Object

edu.uci.jforests.eval.EvaluationMetric

edu.uci.jforests.eval.ranking.RankingEvaluationMetric

edu.uci.jforests.eval.ranking.URiskAwareEval

edu.uci.jforests.eval.ranking.TRiskAwareSAROEval

Direct Known Subclasses:

TRiskAwareFAROEval

public class TRiskAwareSAROEval
extends URiskAwareEval

Implements the TRisk risk-sensitive optimisation metric called SARO, as defined by Dincer et al. See Taner Dincer, Craig Macdonald and Iadh Ounis, Hypothesis Testing for Risk-Sensitive Evaluation of Retrieval Systems, SIGIR 2014.

Implementation Details

It is assumed that the natural ordering of documents for each query defines the baseline.

Nested Class Summary

Nested classes/interfaces inherited from class edu.uci.jforests.eval.ranking.URiskAwareEval

URiskAwareEval.TestURiskSwaps

Nested classes/interfaces inherited from class edu.uci.jforests.eval.ranking.RankingEvaluationMetric

RankingEvaluationMetric.SwapScorer

Constructor Summary

Constructors

Constructor and Description
TRiskAwareSAROEval(EvaluationMetric _parent, double alpha)

Method Summary

Modifier and Type	Method and Description
static double[]	getEstimates(double[] baselinePerQuery, double[] perQuery, double ALPHA) returns double[] params where params[0] = URisk; params[1] = PairedVar.
RankingEvaluationMetric.SwapScorer	getSwapScorer(double[] targets, int[] boundaries, int trunc, int[][] labelCounts)
double	measure(double[] predictions, Sample sample)
static double	T_measure(double[] baselinePerQuery, double[] perQuery, double ALPHA)

Methods inherited from class edu.uci.jforests.eval.ranking.URiskAwareEval

getParentMetric, measureByQuery

Methods inherited from class edu.uci.jforests.eval.ranking.RankingEvaluationMetric

computeNaturalOrderScores

Methods inherited from class edu.uci.jforests.eval.EvaluationMetric

isFirstBetter, largerIsBetter

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TRiskAwareSAROEval

public TRiskAwareSAROEval(EvaluationMetric _parent,
                          double alpha)

Method Detail

getEstimates

public static double[] getEstimates(double[] baselinePerQuery,
                                    double[] perQuery,
                                    double ALPHA)

returns double[] params where params[0] = URisk; params[1] = PairedVar.

T_measure

public static double T_measure(double[] baselinePerQuery,
                               double[] perQuery,
                               double ALPHA)

measure

public double measure(double[] predictions,
                      Sample sample)
               throws java.lang.Exception

Overrides:

measure in class URiskAwareEval

Throws:

java.lang.Exception

getSwapScorer

public RankingEvaluationMetric.SwapScorer getSwapScorer(double[] targets,
                                                        int[] boundaries,
                                                        int trunc,
                                                        int[][] labelCounts)
                                                 throws java.lang.Exception

Overrides:

getSwapScorer in class URiskAwareEval

Parameters:

targets - Targets of all documents in dataset. Equals Dataset.*.length

boundaries - Start and end indices of each query within the dataset.

trunc - Truncation level for evaluation measure

labelCounts - How many of each label value

Throws:

java.lang.Exception - if things go pear shaped