# Statistical Comparisons of Classifiers by Generalized Stochastic Dominance

arXiv: https://arxiv.org/abs/2209.01857 · HTML: https://mlsanity.com/paper/2209.01857/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 67 references that are in the map.
- First posted on arXiv 5 Sep 2022.

## Builds on (most cited first, 8 of 8)

- [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018)
- [Should we really use post-hoc tests based on mean-ranks?](https://mlsanity.com/paper/1505.02288/) (2015)
- [Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis](https://mlsanity.com/paper/1606.04316/) (2016)
- [Veridical Data Science](https://mlsanity.com/paper/1901.08152/) (2019)
- [ASlib: A Benchmark Library for Algorithm Selection](https://mlsanity.com/paper/1506.02465/) (2015)
- [Statistical comparison of classifiers through Bayesian hierarchical modelling](https://mlsanity.com/paper/1609.08905/) (2016)
- [Information efficient learning of complexly structured preferences: Elicitation procedures and their application to decision making under uncertainty](https://mlsanity.com/paper/2110.12879/) (2021)
- [Individualized Decision-Making Under Partial Identification: Three Perspectives, Two Optimality Results, and One Paradox](https://mlsanity.com/paper/2110.10961/) (2021)

## Built on by (most cited first, 20 of 20)

- [An Approach to Multiple Comparison Benchmark Evaluations that is Stable Under Manipulation of the Comparate Set](https://mlsanity.com/paper/2305.11921/) (2023)
- [Robust Statistical Comparison of Random Variables with Locally Varying Scale of Measurement](https://mlsanity.com/paper/2306.12803/) (2023)
- [Explaining Bayesian Optimization by Shapley Values Facilitates Human-AI Collaboration](https://mlsanity.com/paper/2403.04629/) (2024)
- [Statistical Multicriteria Benchmarking via the GSD-Front](https://mlsanity.com/paper/2406.03924/) (2024)
- [Approximately Bayes-Optimal Pseudo Label Selection](https://mlsanity.com/paper/2302.08883/) (2023)
- [Statistical Uncertainty Quantification for Aggregate Performance Metrics in Machine Learning Benchmarks](https://mlsanity.com/paper/2501.04234/) (2025)
- [Towards Better Open-Ended Text Generation: A Multicriteria Evaluation Framework](https://mlsanity.com/paper/2410.18653/) (2024)
- [Statistical Multicriteria Evaluation of LLM-Generated Text](https://mlsanity.com/paper/2506.18082/) (2025)
- [Partial Rankings of Optimizers](https://mlsanity.com/paper/2402.16565/) (2024)
- [Multi-Target Decision Making under Conditions of Severe Uncertainty](https://mlsanity.com/paper/2212.06832/) (2022)
- [Semi-Supervised Learning guided by the Generalized Bayes Rule under Soft Revision](https://mlsanity.com/paper/2405.15294/) (2024)
- [A Statistical Case Against Empirical Human-AI Alignment](https://mlsanity.com/paper/2502.14581/) (2025)
- [Depth Functions for Partial Orders with a Descriptive Analysis of Machine Learning Algorithms](https://mlsanity.com/paper/2304.09872/) (2023)
- [Generalization Bounds and Stopping Rules for Learning with Self-Selected Data](https://mlsanity.com/paper/2505.07367/) (2025)
- [Towards Bayesian Data Selection](https://mlsanity.com/paper/2406.12560/) (2024)
- [Beyond Expectations: Learning with Stochastic Dominance Made Practical](https://mlsanity.com/paper/2402.02698/) (2024)
- [Beyond Arrow: From Impossibility to Possibilities in Multi-Criteria Benchmarking](https://mlsanity.com/paper/2602.07593/) (2026)
- [Reciprocal Learning](https://mlsanity.com/paper/2408.06257/) (2024)
- [Empirical Decision Theory](https://mlsanity.com/paper/2512.05677/) (2025)
- [Self-Supervised Laplace Approximation for Bayesian Uncertainty Quantification](https://mlsanity.com/paper/2605.12208/) (2026)

## Abstract (arXiv)

Although being a crucial question for the development of machine learning algorithms, there is still no consensus on how to compare classifiers over multiple data sets with respect to several criteria. Every comparison framework is confronted with (at least) three fundamental challenges: the multiplicity of quality criteria, the multiplicity of data sets and the randomness of the selection of data sets. In this paper, we add a fresh view to the vivid debate by adopting recent developments in decision theory. Based on so-called preference systems, our framework ranks classifiers by a generalized concept of stochastic dominance, which powerfully circumvents the cumbersome, and often even self-contradictory, reliance on aggregates. Moreover, we show that generalized stochastic dominance can be operationalized by solving easy-to-handle linear programs and moreover statistically tested employing an adapted two-sample observation-randomization test. This yields indeed a powerful framework for the statistical comparison of classifiers over multiple data sets with respect to multiple quality criteria simultaneously. We illustrate and investigate our framework in a simulation study and with a set of standard benchmark data sets.