# A Bayesian Bradley-Terry model to compare multiple ML algorithms on multiple data sets

arXiv: https://arxiv.org/abs/2208.04935 · HTML: https://mlsanity.com/paper/2208.04935/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 62 references that are in the map.
- First posted on arXiv 9 Aug 2022.

## Builds on (most cited first, 14 of 14)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [OpenML Benchmarking Suites](https://mlsanity.com/paper/1708.03731/) (2017)
- [PMLB: A Large Benchmark Suite for Machine Learning Evaluation and Comparison](https://mlsanity.com/paper/1703.00512/) (2017)
- [Should we really use post-hoc tests based on mean-ranks?](https://mlsanity.com/paper/1505.02288/) (2015)
- [Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC](https://mlsanity.com/paper/1507.04544/) (2015)
- [Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis](https://mlsanity.com/paper/1606.04316/) (2016)
- [Evaluation Measures for Hierarchical Classification: a unified view and novel approaches](https://mlsanity.com/paper/1306.6802/) (2013)
- [Nested cross-validation when selecting classifiers is overzealous for most practical applications](https://mlsanity.com/paper/1809.09446/) (2018)
- [The prior can generally only be understood in the context of the likelihood](https://mlsanity.com/paper/1708.07487/) (2017)
- [Statistical comparison of classifiers through Bayesian hierarchical modelling](https://mlsanity.com/paper/1609.08905/) (2016)
- [How to tune the RBF SVM hyperparameters?: An empirical evaluation of 18 search algorithms](https://mlsanity.com/paper/2008.11655/) (2020)
- [Comparison of 14 different families of classification algorithms on 115 binary datasets](https://mlsanity.com/paper/1606.00930/) (2016)
- [Bayesian Paired-Comparison with the bpcs Package](https://mlsanity.com/paper/2101.11227/) (2021)

## Built on by (most cited first, 9 of 9)

- [An Approach to Multiple Comparison Benchmark Evaluations that is Stable Under Manipulation of the Comparate Set](https://mlsanity.com/paper/2305.11921/) (2023)
- [Benchmarking Pretrained Molecular Embedding Models For Molecular Representation Learning](https://mlsanity.com/paper/2508.06199/) (2025)
- [Bayesian Active Learning for Multi-Criteria Comparative Judgement in Educational Assessment](https://mlsanity.com/paper/2503.00479/) (2025)
- [BiasBed -- Rigorous Texture Bias Evaluation](https://mlsanity.com/paper/2211.13190/) (2022)
- [A Bayesian Active Learning Approach to Comparative Judgement](https://mlsanity.com/paper/2308.13292/) (2023)
- [Rendering Transparency to Ranking in Educational Assessment via Bayesian Comparative Judgement](https://mlsanity.com/paper/2503.15549/) (2025)
- [Efficient Portfolio Selection through Preference Aggregation with Quicksort and the Bradley--Terry Model](https://mlsanity.com/paper/2504.16093/) (2025)
- [Mallows Model with Learned Distance Metrics: Sampling and Maximum Likelihood Estimation](https://mlsanity.com/paper/2507.08108/) (2025)
- [Bayesian Preference Learning for Test-Time Steerable Reward Models](https://mlsanity.com/paper/2602.08819/) (2026)

## Abstract (arXiv)

This paper proposes a Bayesian model to compare multiple algorithms on multiple data sets, on any metric. The model is based on the Bradley-Terry model, that counts the number of times one algorithm performs better than another on different data sets. Because of its Bayesian foundations, the Bayesian Bradley Terry model (BBT) has different characteristics than frequentist approaches to comparing multiple algorithms on multiple data sets, such as Demsar (2006) tests on mean rank, and Benavoli et al. (2016) multiple pairwise Wilcoxon tests with p-adjustment procedures. In particular, a Bayesian approach allows for more nuanced statements regarding the algorithms beyond claiming that the difference is or it is not statistically significant. Bayesian approaches also allow to define when two algorithms are equivalent for practical purposes, or the region of practical equivalence (ROPE). Different than a Bayesian signed rank comparison procedure proposed by Benavoli et al. (2017), our approach can define a ROPE for any metric, since it is based on probability statements, and not on differences of that metric. This paper also proposes a local ROPE concept, that evaluates whether a positive difference between a mean measure across some cross validation to the mean of some other algorithms is should be really seen as the first algorithm being better than the second, based on effect sizes. This local ROPE proposal is independent of a Bayesian use, and can be used in frequentist approaches based on ranks. A R package and a Python program that implements the BBT is available.