# Partial Rankings of Optimizers

arXiv: https://arxiv.org/abs/2402.16565 · HTML: https://mlsanity.com/paper/2402.16565/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 39 references that are in the map.
- First posted on arXiv 26 Feb 2024.

## Builds on (most cited first, 11 of 11)

- [COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting](https://mlsanity.com/paper/1603.08785/) (2016)
- [MLPerf Training Benchmark](https://mlsanity.com/paper/1910.01500/) (2019)
- [PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithm](https://mlsanity.com/paper/2208.07914/) (2022)
- [Benchmarking Neural Network Training Algorithms](https://mlsanity.com/paper/2306.07179/) (2023)
- [DeepOBS: A Deep Learning Optimizer Benchmark Suite](https://mlsanity.com/paper/1903.05499/) (2019)
- [Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RL](https://mlsanity.com/paper/2305.00567/) (2023)
- [Statistical Comparisons of Classifiers by Generalized Stochastic Dominance](https://mlsanity.com/paper/2209.01857/) (2022)
- [Choosing among notions of multivariate depth statistics](https://mlsanity.com/paper/2004.01927/) (2020)
- [Robust Statistical Comparison of Random Variables with Locally Varying Scale of Measurement](https://mlsanity.com/paper/2306.12803/) (2023)
- [Min-Max Bilevel Multi-objective Optimization with Applications in Machine Learning](https://mlsanity.com/paper/2203.01924/) (2022)
- [Depth Functions for Partial Orders with a Descriptive Analysis of Machine Learning Algorithms](https://mlsanity.com/paper/2304.09872/) (2023)

## Built on by (most cited first, 8 of 8)

- [Explaining Bayesian Optimization by Shapley Values Facilitates Human-AI Collaboration](https://mlsanity.com/paper/2403.04629/) (2024)
- [Statistical Multicriteria Benchmarking via the GSD-Front](https://mlsanity.com/paper/2406.03924/) (2024)
- [Statistical Multicriteria Evaluation of LLM-Generated Text](https://mlsanity.com/paper/2506.18082/) (2025)
- [A Statistical Case Against Empirical Human-AI Alignment](https://mlsanity.com/paper/2502.14581/) (2025)
- [Beyond Arrow: From Impossibility to Possibilities in Multi-Criteria Benchmarking](https://mlsanity.com/paper/2602.07593/) (2026)
- [Reciprocal Learning](https://mlsanity.com/paper/2408.06257/) (2024)
- [Regime-Conditioned Evaluation in Multi-Context Bayesian Optimization](https://mlsanity.com/paper/2605.04895/) (2026)
- [Self-Supervised Laplace Approximation for Bayesian Uncertainty Quantification](https://mlsanity.com/paper/2605.12208/) (2026)

## Abstract (arXiv)

We introduce a framework for benchmarking optimizers according to multiple criteria over various test functions. Based on a recently introduced union-free generic depth function for partial orders/rankings, it fully exploits the ordinal information and allows for incomparability. Our method describes the distribution of all partial orders/rankings, avoiding the notorious shortcomings of aggregation. This permits to identify test functions that produce central or outlying rankings of optimizers and to assess the quality of benchmarking suites.