# Foundations of the Theory of Performance-Based Ranking

arXiv: https://arxiv.org/abs/2412.04227 · HTML: https://mlsanity.com/paper/2412.04227/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 43 references that are in the map.
- First posted on arXiv 5 Dec 2024.

## Builds on (most cited first, 4 of 4)

- [Why rankings of biomedical image analysis competitions should be interpreted with care](https://mlsanity.com/paper/1806.02051/) (2018)
- [SoccerNet 2023 Challenges Results](https://mlsanity.com/paper/2309.06006/) (2023)
- [How Trustworthy are Performance Evaluations for Basic Vision Tasks?](https://mlsanity.com/paper/2008.03533/) (2020)
- [Summarizing the performances of a background subtraction algorithm measured on several videos](https://mlsanity.com/paper/2002.05654/) (2020)

## Built on by (most cited first, 3 of 3)

- [A Methodology to Evaluate Strategies Predicting Rankings on Unseen Domains](https://mlsanity.com/paper/2505.15595/) (2025)
- [LinDeps: A Fine-tuning Free Post-Pruning Method to Remove Layer-Wise Linear Dependencies with Guaranteed Performance Preservation](https://mlsanity.com/paper/2507.21573/) (2025)
- [Multi-domain performance analysis with scores tailored to user preferences](https://mlsanity.com/paper/2512.08715/) (2025)

## Abstract (arXiv)

Ranking entities such as algorithms, devices, methods, or models based on their performances, while accounting for application-specific preferences, is a challenge. To address this challenge, we establish the foundations of a universal theory for performance-based ranking. First, we introduce a rigorous framework built on top of both the probability and order theories. Our new framework encompasses the elements necessary to (1) manipulate performances as mathematical objects, (2) express which performances are worse than or equivalent to others, (3) model tasks through a variable called satisfaction, (4) consider properties of the evaluation, (5) define scores, and (6) specify application-specific preferences through a variable called importance. On top of this framework, we propose the first axiomatic definition of performance orderings and performance-based rankings. Then, we introduce a universal parametric family of scores, called ranking scores, that can be used to establish rankings satisfying our axioms, while considering application-specific preferences. Finally, we show, in the case of two-class classification, that the family of ranking scores encompasses well-known performance scores, including the accuracy, the true positive rate (recall, sensitivity), the true negative rate (specificity), the positive predictive value (precision), and F1. However, we also show that some other scores commonly used to compare classifiers are unsuitable to derive performance orderings satisfying the axioms.