# Properties of Group Fairness Metrics for Rankings

arXiv: https://arxiv.org/abs/2212.14351 · HTML: https://mlsanity.com/paper/2212.14351/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 58 references that are in the map.
- First posted on arXiv 29 Dec 2022.

## Builds on (most cited first, 21 of 21)

- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Fairness of Exposure in Rankings](https://mlsanity.com/paper/1802.07281/) (2018)
- [Fairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Search](https://mlsanity.com/paper/1905.01989/) (2019)
- [Fairness in Recommendation Ranking through Pairwise Comparisons](https://mlsanity.com/paper/1903.00780/) (2019)
- [Equity of Attention: Amortizing Individual Fairness in Rankings](https://mlsanity.com/paper/1805.01788/) (2018)
- [A Theoretical Analysis of NDCG Type Ranking Measures](https://mlsanity.com/paper/1304.6480/) (2013)
- [A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learning](https://mlsanity.com/paper/2203.11933/) (2022)
- [Measuring Fairness in Ranked Outputs](https://mlsanity.com/paper/1610.08559/) (2016)
- [Ranking with Fairness Constraints](https://mlsanity.com/paper/1704.06840/) (2017)
- [Policy Learning for Fairness in Ranking](https://mlsanity.com/paper/1902.04056/) (2019)
- [Controlling Fairness and Bias in Dynamic Learning-to-Rank](https://mlsanity.com/paper/2005.14713/) (2020)
- [Fairness in Rankings and Recommendations: An Overview](https://mlsanity.com/paper/2104.05994/) (2021)
- [Pairwise Fairness for Ranking and Regression](https://mlsanity.com/paper/1906.05330/) (2019)
- [Evaluating Stochastic Rankings with Expected Exposure](https://mlsanity.com/paper/2004.13157/) (2020)
- [Interventions for Ranking in the Presence of Implicit Bias](https://mlsanity.com/paper/2001.08767/) (2020)
- [Quantifying the Impact of User Attention on Fair Group Representation in Ranked Lists](https://mlsanity.com/paper/1901.10437/) (2019)
- [When Fair Ranking Meets Uncertain Inference](https://mlsanity.com/paper/2105.02091/) (2021)
- [On Discrimination Discovery and Removal in Ranked Data using Causal Graph](https://mlsanity.com/paper/1803.01901/) (2018)
- [Assessing Viewpoint Diversity in Search Results Using Ranking Fairness Metrics](https://mlsanity.com/paper/2010.14531/) (2020)
- [An Axiomatic Analysis of Diversity Evaluation Metrics: Introducing the Rank-Biased Utility Metric](https://mlsanity.com/paper/1805.02334/) (2018)
- [Evaluating Fairness in Argument Retrieval](https://mlsanity.com/paper/2108.10442/) (2021)

## Built on by (most cited first, 2 of 2)

- [Stairway to Fairness: Connecting Group and Individual Fairness](https://mlsanity.com/paper/2508.21334/) (2025)
- [Controllable Pareto Trade-off between Fairness and Accuracy](https://mlsanity.com/paper/2509.13651/) (2025)

## Abstract (arXiv)

In recent years, several metrics have been developed for evaluating group fairness of rankings. Given that these metrics were developed with different application contexts and ranking algorithms in mind, it is not straightforward which metric to choose for a given scenario. In this paper, we perform a comprehensive comparative analysis of existing group fairness metrics developed in the context of fair ranking. By virtue of their diverse application contexts, we argue that such a comparative analysis is not straightforward. Hence, we take an axiomatic approach whereby we design a set of thirteen properties for group fairness metrics that consider different ranking settings. A metric can then be selected depending on whether it satisfies all or a subset of these properties. We apply these properties on eleven existing group fairness metrics, and through both empirical and theoretical results we demonstrate that most of these metrics only satisfy a small subset of the proposed properties. These findings highlight limitations of existing metrics, and provide insights into how to evaluate and interpret different fairness metrics in practical deployment. The proposed properties can also assist practitioners in selecting appropriate metrics for evaluating fairness in a specific application.