# The Role of Relevance in Fair Ranking

arXiv: https://arxiv.org/abs/2305.05608 · HTML: https://mlsanity.com/paper/2305.05608/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 81 references that are in the map.
- First posted on arXiv 9 May 2023.

## Builds on (most cited first, 30 of 33)

- [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019)
- [Recommendations as Treatments: Debiasing Learning and Evaluation](https://mlsanity.com/paper/1602.05352/) (2016)
- [Fairness of Exposure in Rankings](https://mlsanity.com/paper/1802.07281/) (2018)
- [Unbiased Learning-to-Rank with Biased Feedback](https://mlsanity.com/paper/1608.04468/) (2016)
- [Fairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Search](https://mlsanity.com/paper/1905.01989/) (2019)
- [Fairness in Recommendation Ranking through Pairwise Comparisons](https://mlsanity.com/paper/1903.00780/) (2019)
- [Equity of Attention: Amortizing Individual Fairness in Rankings](https://mlsanity.com/paper/1805.01788/) (2018)
- [Policy Learning for Fairness in Ranking](https://mlsanity.com/paper/1902.04056/) (2019)
- [Controlling Fairness and Bias in Dynamic Learning-to-Rank](https://mlsanity.com/paper/2005.14713/) (2020)
- [Learning a Deep Listwise Context Model for Ranking Refinement](https://mlsanity.com/paper/1804.05936/) (2018)
- [Unbiased Learning to Rank with Unbiased Propensity Estimation](https://mlsanity.com/paper/1804.05938/) (2018)
- [Evaluating Stochastic Rankings with Expected Exposure](https://mlsanity.com/paper/2004.13157/) (2020)
- [Designing Fair Ranking Schemes](https://mlsanity.com/paper/1712.09752/) (2017)
- [Fairness in Ranking: A Survey](https://mlsanity.com/paper/2103.14000/) (2021)
- [Fairness in Information Access Systems](https://mlsanity.com/paper/2105.05779/) (2021)
- [CPFair: Personalized Consumer and Producer Fairness Re-ranking for Recommender Systems](https://mlsanity.com/paper/2204.08085/) (2022)
- [MS MARCO: Benchmarking Ranking Models in the Large-Data Regime](https://mlsanity.com/paper/2105.04021/) (2021)
- [Differentiable Unbiased Online Learning to Rank](https://mlsanity.com/paper/1809.08415/) (2018)
- [Quantifying the Impact of User Attention on Fair Group Representation in Ranked Lists](https://mlsanity.com/paper/1901.10437/) (2019)
- [When Fair Ranking Meets Uncertain Inference](https://mlsanity.com/paper/2105.02091/) (2021)
- [Fair ranking: a critical review, challenges, and future directions](https://mlsanity.com/paper/2201.12662/) (2022)
- [Cascade Model-based Propensity Estimation for Counterfactual Learning to Rank](https://mlsanity.com/paper/2005.11938/) (2020)
- [Estimation of Fair Ranking Metrics with Incomplete Judgments](https://mlsanity.com/paper/2108.05152/) (2021)
- [Causal intersectionality for fair ranking](https://mlsanity.com/paper/2006.08688/) (2020)
- [Exploring Artist Gender Bias in Music Recommendation](https://mlsanity.com/paper/2009.01715/) (2020)
- [Maximizing Marginal Fairness for Dynamic Learning to Rank](https://mlsanity.com/paper/2102.09670/) (2021)
- [Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank](https://mlsanity.com/paper/2206.12204/) (2022)
- [Overview of the TREC 2021 Fair Ranking Track](https://mlsanity.com/paper/2302.10856/) (2023)
- [What is fair? Exploring the artists' perspective on the fairness of music streaming platforms](https://mlsanity.com/paper/2106.02415/) (2021)
- [Overview of the TREC 2020 Fair Ranking Track](https://mlsanity.com/paper/2108.05135/) (2021)

## Built on by (most cited first, 3 of 3)

- [Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generation](https://mlsanity.com/paper/2409.11598/) (2024)
- [What's in a Query: Polarity-Aware Distribution-Based Fair Ranking](https://mlsanity.com/paper/2502.11429/) (2025)
- [A Tutorial On Intersectionality in Fair Rankings](https://mlsanity.com/paper/2502.05333/) (2025)

## Abstract (arXiv)

Online platforms mediate access to opportunity: relevance-based rankings create and constrain options by allocating exposure to job openings and job candidates in hiring platforms, or sellers in a marketplace. In order to do so responsibly, these socially consequential systems employ various fairness measures and interventions, many of which seek to allocate exposure based on worthiness. Because these constructs are typically not directly observable, platforms must instead resort to using proxy scores such as relevance and infer them from behavioral signals such as searcher clicks. Yet, it remains an open question whether relevance fulfills its role as such a worthiness score in high-stakes fair rankings. In this paper, we combine perspectives and tools from the social sciences, information retrieval, and fairness in machine learning to derive a set of desired criteria that relevance scores should satisfy in order to meaningfully guide fairness interventions. We then empirically show that not all of these criteria are met in a case study of relevance inferred from biased user click data. We assess the impact of these violations on the estimated system fairness and analyze whether existing fairness interventions may mitigate the identified issues. Our analyses and results surface the pressing need for new approaches to relevance collection and generation that are suitable for use in fair ranking.