# Bipartite Ranking From Multiple Labels: On Loss Versus Label Aggregation

arXiv: https://arxiv.org/abs/2504.11284 · HTML: https://mlsanity.com/paper/2504.11284/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 56 references that are in the map.
- First posted on arXiv 15 Apr 2025.

## Builds on (most cited first, 16 of 16)

- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Introducing LETOR 4.0 Datasets](https://mlsanity.com/paper/1306.2597/) (2013)
- [A Theoretical Analysis of NDCG Type Ranking Measures](https://mlsanity.com/paper/1304.6480/) (2013)
- [Fairness risk measures](https://mlsanity.com/paper/1901.08665/) (2019)
- [Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensembles](https://mlsanity.com/paper/2210.16955/) (2022)
- [Scalable Pareto Front Approximation for Deep Multi-Objective Learning](https://mlsanity.com/paper/2103.13392/) (2021)
- [TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank](https://mlsanity.com/paper/1812.00073/) (2018)
- [To Aggregate or Not? Learning with Separate Noisy Labels](https://mlsanity.com/paper/2206.07181/) (2022)
- [Multi-Label Learning to Rank through Multi-Objective Optimization](https://mlsanity.com/paper/2207.03060/) (2022)
- [Top Rank Optimization in Linear Time](https://mlsanity.com/paper/1410.1462/) (2014)
- [Rethinking and Reweighting the Univariate Losses for Multi-Label Ranking: Consistency and Generalization](https://mlsanity.com/paper/2105.05026/) (2021)
- [Que2Engage: Embedding-based Retrieval for Relevant and Engaging Products at Facebook Marketplace](https://mlsanity.com/paper/2302.11052/) (2023)
- [Towards the Better Ranking Consistency: A Multi-task Learning Framework for Early Stage Ads Ranking](https://mlsanity.com/paper/2307.11096/) (2023)
- [Multicriteria Optimization and Decision Making: Principles, Algorithms and Case Studies](https://mlsanity.com/paper/2407.00359/) (2024)
- [Enhancing Personalized Ranking With Differentiable Group AUC Optimization](https://mlsanity.com/paper/2304.09176/) (2023)

## Built on by (most cited first, 2 of 2)

- [Establishing Linear Surrogate Regret Bounds for Convex Smooth Losses via Convolutional Fenchel-Young Losses](https://mlsanity.com/paper/2505.09432/) (2025)
- [ACES: Who Tests the Tests? Leave-One-Out AUC Consistency for Code Generation](https://mlsanity.com/paper/2604.03922/) (2026)

## Abstract (arXiv)

Bipartite ranking is a fundamental supervised learning problem, with the goal of learning a ranking over instances with maximal Area Under the ROC Curve (AUC) against a single binary target label. However, one may often observe multiple binary target labels, e.g., from distinct human annotators. How can one synthesize such labels into a single coherent ranking? In this work, we formally analyze two approaches to this problem -- loss aggregation and label aggregation -- by characterizing their Bayes-optimal solutions. We show that while both approaches can yield Pareto-optimal solutions, loss aggregation can exhibit label dictatorship: one can inadvertently (and undesirably) favor one label over others. This suggests that label aggregation can be preferable to loss aggregation, which we empirically verify.