mlsanityData to 14 Oct 2026

Statistical Multicriteria Benchmarking via the GSD-Front

arXiv 2406.03924 ↗First posted 6 Jun 2024stat.ML cs.LG stat.MEcited by 11

Not on the map. Why?

Abstract (arXiv)

Given the vast number of classifiers that have been (and continue to be) proposed, reliable methods for comparing them are becoming increasingly important. The desire for reliability is broken down into three main aspects: (1) Comparisons should allow for different quality metrics simultaneously. (2) Comparisons should take into account the statistical uncertainty induced by the choice of benchmark suite. (3) The robustness of the comparisons under small deviations in the underlying assumptions should be verifiable. To address (1), we propose to compare classifiers using a generalized stochastic dominance ordering (GSD) and present the GSD-front as an information-efficient alternative to the classical Pareto-front. For (2), we propose a consistent statistical estimator for the GSD-front and construct a statistical test for whether a (potentially new) classifier lies in the GSD-front of a set of state-of-the-art classifiers. For (3), we relax our proposed test using techniques from robust statistics and imprecise probabilities. We illustrate our concepts on the benchmark suite PMLB and on the platform OpenML.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 28 of its 86 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

28 of 86 references.

References · 28

  1. 2018Deep learning for time series classification: a reviewcited by 576
  2. 2019Machine Learning Testing: Survey, Landscapes and Horizonscited by 218
  3. 2015ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and Rcited by 172
  4. 2017PMLB: A Large Benchmark Suite for Machine Learning Evaluation and Comparisoncited by 140
  5. 2015Should we really use post-hoc tests based on mean-ranks?cited by 116
  6. 2016Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysiscited by 93
  7. 2022Mapping global dynamics of benchmark creation and saturation in artificial intelligencecited by 71
  8. 2019Veridical Data Sciencecited by 66
  9. 2015ASlib: A Benchmark Library for Algorithm Selectioncited by 37
  10. 2019DeepOBS: A Deep Learning Optimizer Benchmark Suitecited by 37
  11. 2024Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarkscited by 28
  12. 2019Quantifying Model Complexity via Functional Decomposition for Better Post-Hoc Interpretabilitycited by 22
16 more
  1. 2022Statistical Comparisons of Classifiers by Generalized Stochastic Dominancecited by 20
  2. 2022A Theory of Dynamic Benchmarkscited by 16
  3. 2021Credal Self-Supervised Learningcited by 15
  4. 2016Statistical comparison of classifiers through Bayesian hierarchical modellingcited by 14
  5. 2021Accounting for Gaussian Process Imprecision in Bayesian Optimizationcited by 13
  6. 2023Robust Statistical Comparison of Random Variables with Locally Varying Scale of Measurementcited by 13
  7. 2023A Novel Bayes' Theorem for Upper Probabilitiescited by 13
  8. 2023Distributionally Robust Statistical Verification with Imprecise Neural Networkscited by 12
  9. 2021Information efficient learning of complexly structured preferences: Elicitation procedures and their application to decision making under uncertaintycited by 10
  10. 2022Attention-based Random Forest and Contamination Modelcited by 10
  11. 2024Partial Rankings of Optimizerscited by 8
  12. 2022Multi-Target Decision Making under Conditions of Severe Uncertaintycited by 7
  13. 2022Learning to Optimize with Stochastic Dominance Constraintscited by 5
  14. 2023Depth Functions for Partial Orders with a Descriptive Analysis of Machine Learning Algorithmscited by 5
  15. 2023Comparing Machine Learning Algorithms by Union-Free Generic Depthcited by 4
  16. 2021On the Assessment of Benchmark Suites for Algorithm Comparisoncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 3.

11 papers.

  1. 2025G2: Guided Generation for Enhanced Output Diversity in LLMscited by 14
  2. 2024Towards Better Open-Ended Text Generation: A Multicriteria Evaluation Frameworkcited by 9
  3. 2025Statistical Multicriteria Evaluation of LLM-Generated Textcited by 9
  4. 2025A Statistical Case Against Empirical Human-AI Alignmentcited by 6
  5. 2025Generalization Bounds and Stopping Rules for Learning with Self-Selected Datacited by 5
  6. 2025FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learningcited by 4
  7. 2026Beyond Arrow: From Impossibility to Possibilities in Multi-Criteria Benchmarkingcited by 1
  8. 2024Reciprocal Learningcited by 0
  9. 2025Empirical Decision Theorycited by 0
  10. 2026Incentive Aware AI Regulations: A Credal Characterisationcited by 0
  11. 2026Self-Supervised Laplace Approximation for Bayesian Uncertainty Quantificationcited by 0