# Bradley-Terry Rankings for Recommender Systems Across Dataset Taxonomies

arXiv: https://arxiv.org/abs/2606.07492 · HTML: https://mlsanity.com/paper/2606.07492/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 49 references that are in the map.
- First posted on arXiv 5 Jun 2026.

## Builds on (most cited first, 21 of 21)

- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Embarrassingly Shallow Autoencoders for Sparse Data](https://mlsanity.com/paper/1905.03375/) (2019)
- [Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis](https://mlsanity.com/paper/1606.04316/) (2016)
- [UltraGCN: Ultra Simplification of Graph Convolutional Networks for Recommendation](https://mlsanity.com/paper/2110.15114/) (2021)
- [AMLB: an AutoML Benchmark](https://mlsanity.com/paper/2207.12560/) (2022)
- [Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives](https://mlsanity.com/paper/2502.02723/) (2025)
- [Uncertainty quantification in the Bradley-Terry-Luce model](https://mlsanity.com/paper/2110.03874/) (2021)
- [Approximate Ranking from Pairwise Comparisons](https://mlsanity.com/paper/1801.01253/) (2018)
- [Uncertainty Quantification of MLE for Entity Ranking with Covariates](https://mlsanity.com/paper/2212.09961/) (2022)
- [Does It Look Sequential? An Analysis of Datasets for Evaluation of Sequential Recommendations](https://mlsanity.com/paper/2408.12008/) (2024)
- [Scalable Cross-Entropy Loss for Sequential Recommendations with Large Item Catalogs](https://mlsanity.com/paper/2409.18721/) (2024)
- [From Variability to Stability: Advancing RecSys Benchmarking Practices](https://mlsanity.com/paper/2402.09766/) (2024)
- [Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders](https://mlsanity.com/paper/2507.16289/) (2025)
- [Tensor-based Sequential Learning via Hankel Matrix Representation for Next Item Recommendations](https://mlsanity.com/paper/2212.05720/) (2022)
- [Paired comparison models with strength-dependent ties and order effects](https://mlsanity.com/paper/2505.24783/) (2025)
- [APS Explorer: Navigating Algorithm Performance Spaces for Informed Dataset Selection](https://mlsanity.com/paper/2508.19399/) (2025)
- [Beyond Pairwise Learning-To-Rank At Airbnb](https://mlsanity.com/paper/2505.09795/) (2025)
- [Matrix-Free Two-to-Infinity and One-to-Two Norms Estimation](https://mlsanity.com/paper/2508.04444/) (2025)

## Abstract (arXiv)

The ranking of recommendation algorithms is a challenging problem since model performance is sensitive to dataset characteristics such as sparsity, sequential structure, and scale. This drives a demand for a proper methodology for fair comparison between algorithms. Naive aggregation of performance metrics (e.g., averaging NDCG over benchmarks) can yield misleading rankings, undermining practical selection. To address this problem, we introduce a novel, data-driven ranking methodology based on Bradley-Terry (BT) model. We demonstrate that the obtained ranking depends on key dataset statistics. Additionally, we propose a novel metric for evaluating ranking consistency and demonstrate robustness of our ranking to incomplete data. Finally, we introduce a dataset-specific methodology for ranking algorithms on unseen datasets without running the models, relying on extensions of the Bradley-Terry framework, including BT trees and BT models with covariates.