# Estimation of Fair Ranking Metrics with Incomplete Judgments

arXiv: https://arxiv.org/abs/2108.05152 · HTML: https://mlsanity.com/paper/2108.05152/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 45 references that are in the map.
- First posted on arXiv 11 Aug 2021.

## Builds on (most cited first, 16 of 16)

- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Fairness Without Demographics in Repeated Loss Minimization](https://mlsanity.com/paper/1806.08010/) (2018)
- [Improving fairness in machine learning systems: What do industry practitioners need?](https://mlsanity.com/paper/1812.05239/) (2018)
- [Fairness without Demographics through Adversarially Reweighted Learning](https://mlsanity.com/paper/2006.13114/) (2020)
- [Fairness of Exposure in Rankings](https://mlsanity.com/paper/1802.07281/) (2018)
- [Fairness in Recommendation Ranking through Pairwise Comparisons](https://mlsanity.com/paper/1903.00780/) (2019)
- [Equity of Attention: Amortizing Individual Fairness in Rankings](https://mlsanity.com/paper/1805.01788/) (2018)
- [FA*IR: A Fair Top-k Ranking Algorithm](https://mlsanity.com/paper/1706.06368/) (2017)
- [Measuring Fairness in Ranked Outputs](https://mlsanity.com/paper/1610.08559/) (2016)
- [Assessing Algorithmic Fairness with Unobserved Protected Class Using Data Combination](https://mlsanity.com/paper/1906.00285/) (2019)
- [Ranking with Fairness Constraints](https://mlsanity.com/paper/1704.06840/) (2017)
- [Multisided Fairness for Recommendation](https://mlsanity.com/paper/1707.00093/) (2017)
- [Controlling Fairness and Bias in Dynamic Learning-to-Rank](https://mlsanity.com/paper/2005.14713/) (2020)
- [Evaluating Stochastic Rankings with Expected Exposure](https://mlsanity.com/paper/2004.13157/) (2020)
- [Exploring Author Gender in Book Rating and Recommendation](https://mlsanity.com/paper/1808.07586/) (2018)
- [Overview of the TREC 2019 Fair Ranking Track](https://mlsanity.com/paper/2003.11650/) (2020)

## Built on by (most cited first, 14 of 14)

- [Fairness in Recommender Systems: Research Landscape and Future Directions](https://mlsanity.com/paper/2205.11127/) (2022)
- [CPFair: Personalized Consumer and Producer Fairness Re-ranking for Recommender Systems](https://mlsanity.com/paper/2204.08085/) (2022)
- [Understanding Biases in ChatGPT-based Recommender Systems: Provider Fairness, Temporal Stability, and Recency](https://mlsanity.com/paper/2401.10545/) (2024)
- [Fair Ranking with Noisy Protected Attributes](https://mlsanity.com/paper/2211.17067/) (2022)
- [Measuring Fairness Under Unawareness of Sensitive Attributes: A Quantification-Based Approach](https://mlsanity.com/paper/2109.08549/) (2021)
- [Re-thinking Knowledge Graph Completion Evaluation from an Information Retrieval Perspective](https://mlsanity.com/paper/2205.04105/) (2022)
- [Simpson's Paradox in Recommender Fairness: Reconciling differences between per-user and aggregated evaluations](https://mlsanity.com/paper/2210.07755/) (2022)
- [FairRoad: Achieving Fairness for Recommender Systems with Optimized Antidote Data](https://mlsanity.com/paper/2212.06750/) (2022)
- [Probabilistic Permutation Graph Search: Black-Box Optimization for Fairness in Ranking](https://mlsanity.com/paper/2204.13765/) (2022)
- [The Role of Relevance in Fair Ranking](https://mlsanity.com/paper/2305.05608/) (2023)
- [A Personalized Framework for Consumer and Producer Group Fairness Optimization in Recommender Systems](https://mlsanity.com/paper/2402.00485/) (2024)
- [Hidden or Inferred: Fair Learning-To-Rank with Unknown Demographics](https://mlsanity.com/paper/2407.17459/) (2024)
- [Statistically Testing Training Data for Unwanted Error Patterns using Rule-Oriented Regression](https://mlsanity.com/paper/2503.18497/) (2025)
- [Exposure-Based Reinforcement Learning to Rank](https://mlsanity.com/paper/2607.18689/) (2026)

## Abstract (arXiv)

There is increasing attention to evaluating the fairness of search system ranking decisions. These metrics often consider the membership of items to particular groups, often identified using protected attributes such as gender or ethnicity. To date, these metrics typically assume the availability and completeness of protected attribute labels of items. However, the protected attributes of individuals are rarely present, limiting the application of fair ranking metrics in large scale systems. In order to address this problem, we propose a sampling strategy and estimation technique for four fair ranking metrics. We formulate a robust and unbiased estimator which can operate even with very limited number of labeled items. We evaluate our approach using both simulated and real world data. Our experimental results demonstrate that our method can estimate this family of fair ranking metrics and provides a robust, reliable alternative to exhaustive or random data annotation.