# Statistical Multicriteria Benchmarking via the GSD-Front

arXiv: https://arxiv.org/abs/2406.03924 · HTML: https://mlsanity.com/paper/2406.03924/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 86 references that are in the map.
- First posted on arXiv 6 Jun 2024.

## Builds on (most cited first, 28 of 28)

- [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R](https://mlsanity.com/paper/1508.04409/) (2015)
- [PMLB: A Large Benchmark Suite for Machine Learning Evaluation and Comparison](https://mlsanity.com/paper/1703.00512/) (2017)
- [Should we really use post-hoc tests based on mean-ranks?](https://mlsanity.com/paper/1505.02288/) (2015)
- [Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis](https://mlsanity.com/paper/1606.04316/) (2016)
- [Mapping global dynamics of benchmark creation and saturation in artificial intelligence](https://mlsanity.com/paper/2203.04592/) (2022)
- [Veridical Data Science](https://mlsanity.com/paper/1901.08152/) (2019)
- [ASlib: A Benchmark Library for Algorithm Selection](https://mlsanity.com/paper/1506.02465/) (2015)
- [DeepOBS: A Deep Learning Optimizer Benchmark Suite](https://mlsanity.com/paper/1903.05499/) (2019)
- [Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarks](https://mlsanity.com/paper/2405.01719/) (2024)
- [Quantifying Model Complexity via Functional Decomposition for Better Post-Hoc Interpretability](https://mlsanity.com/paper/1904.03867/) (2019)
- [Statistical Comparisons of Classifiers by Generalized Stochastic Dominance](https://mlsanity.com/paper/2209.01857/) (2022)
- [A Theory of Dynamic Benchmarks](https://mlsanity.com/paper/2210.03165/) (2022)
- [Credal Self-Supervised Learning](https://mlsanity.com/paper/2106.11853/) (2021)
- [Statistical comparison of classifiers through Bayesian hierarchical modelling](https://mlsanity.com/paper/1609.08905/) (2016)
- [Accounting for Gaussian Process Imprecision in Bayesian Optimization](https://mlsanity.com/paper/2111.08299/) (2021)
- [Robust Statistical Comparison of Random Variables with Locally Varying Scale of Measurement](https://mlsanity.com/paper/2306.12803/) (2023)
- [A Novel Bayes' Theorem for Upper Probabilities](https://mlsanity.com/paper/2307.06831/) (2023)
- [Distributionally Robust Statistical Verification with Imprecise Neural Networks](https://mlsanity.com/paper/2308.14815/) (2023)
- [Information efficient learning of complexly structured preferences: Elicitation procedures and their application to decision making under uncertainty](https://mlsanity.com/paper/2110.12879/) (2021)
- [Attention-based Random Forest and Contamination Model](https://mlsanity.com/paper/2201.02880/) (2022)
- [Partial Rankings of Optimizers](https://mlsanity.com/paper/2402.16565/) (2024)
- [Multi-Target Decision Making under Conditions of Severe Uncertainty](https://mlsanity.com/paper/2212.06832/) (2022)
- [Learning to Optimize with Stochastic Dominance Constraints](https://mlsanity.com/paper/2211.07767/) (2022)
- [Depth Functions for Partial Orders with a Descriptive Analysis of Machine Learning Algorithms](https://mlsanity.com/paper/2304.09872/) (2023)
- [Comparing Machine Learning Algorithms by Union-Free Generic Depth](https://mlsanity.com/paper/2312.12839/) (2023)
- [On the Assessment of Benchmark Suites for Algorithm Comparison](https://mlsanity.com/paper/2104.07381/) (2021)

## Built on by (most cited first, 11 of 11)

- [G2: Guided Generation for Enhanced Output Diversity in LLMs](https://mlsanity.com/paper/2511.00432/) (2025)
- [Towards Better Open-Ended Text Generation: A Multicriteria Evaluation Framework](https://mlsanity.com/paper/2410.18653/) (2024)
- [Statistical Multicriteria Evaluation of LLM-Generated Text](https://mlsanity.com/paper/2506.18082/) (2025)
- [A Statistical Case Against Empirical Human-AI Alignment](https://mlsanity.com/paper/2502.14581/) (2025)
- [Generalization Bounds and Stopping Rules for Learning with Self-Selected Data](https://mlsanity.com/paper/2505.07367/) (2025)
- [FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2510.20250/) (2025)
- [Beyond Arrow: From Impossibility to Possibilities in Multi-Criteria Benchmarking](https://mlsanity.com/paper/2602.07593/) (2026)
- [Reciprocal Learning](https://mlsanity.com/paper/2408.06257/) (2024)
- [Empirical Decision Theory](https://mlsanity.com/paper/2512.05677/) (2025)
- [Incentive Aware AI Regulations: A Credal Characterisation](https://mlsanity.com/paper/2603.05175/) (2026)
- [Self-Supervised Laplace Approximation for Bayesian Uncertainty Quantification](https://mlsanity.com/paper/2605.12208/) (2026)

## Abstract (arXiv)

Given the vast number of classifiers that have been (and continue to be) proposed, reliable methods for comparing them are becoming increasingly important. The desire for reliability is broken down into three main aspects: (1) Comparisons should allow for different quality metrics simultaneously. (2) Comparisons should take into account the statistical uncertainty induced by the choice of benchmark suite. (3) The robustness of the comparisons under small deviations in the underlying assumptions should be verifiable. To address (1), we propose to compare classifiers using a generalized stochastic dominance ordering (GSD) and present the GSD-front as an information-efficient alternative to the classical Pareto-front. For (2), we propose a consistent statistical estimator for the GSD-front and construct a statistical test for whether a (potentially new) classifier lies in the GSD-front of a set of state-of-the-art classifiers. For (3), we relax our proposed test using techniques from robust statistics and imprecise probabilities. We illustrate our concepts on the benchmark suite PMLB and on the platform OpenML.