# De-biasing "bias" measurement

arXiv: https://arxiv.org/abs/2205.05770 · HTML: https://mlsanity.com/paper/2205.05770/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 54 references that are in the map.
- First posted on arXiv 11 May 2022.

## Builds on (most cited first, 25 of 25)

- [Mitigating Unwanted Biases with Adversarial Learning](https://mlsanity.com/paper/1801.07593/) (2018)
- [A Reductions Approach to Fair Classification](https://mlsanity.com/paper/1803.02453/) (2018)
- [Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness](https://mlsanity.com/paper/1711.05144/) (2017)
- [Fairness Without Demographics in Repeated Loss Minimization](https://mlsanity.com/paper/1806.08010/) (2018)
- [Learning Adversarially Fair and Transferable Representations](https://mlsanity.com/paper/1802.06309/) (2018)
- [AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias](https://mlsanity.com/paper/1810.01943/) (2018)
- [A comparative study of fairness-enhancing interventions in machine learning](https://mlsanity.com/paper/1802.04422/) (2018)
- [Delayed Impact of Fair Machine Learning](https://mlsanity.com/paper/1803.04383/) (2018)
- [Aequitas: A Bias and Fairness Audit Toolkit](https://mlsanity.com/paper/1811.05577/) (2018)
- [Fairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Search](https://mlsanity.com/paper/1905.01989/) (2019)
- [FA*IR: A Fair Top-k Ranking Algorithm](https://mlsanity.com/paper/1706.06368/) (2017)
- [A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual & Group Unfairness via Inequality Indices](https://mlsanity.com/paper/1807.00787/) (2018)
- [FairVis: Visual Analytics for Discovering Intersectional Bias in Machine Learning](https://mlsanity.com/paper/1904.05419/) (2019)
- [The Disparate Effects of Strategic Manipulation](https://mlsanity.com/paper/1808.08646/) (2018)
- [Training individually fair ML models with Sensitive Subspace Robustness](https://mlsanity.com/paper/1907.00020/) (2019)
- [FairSight: Visual Analytics for Fairness in Decision Making](https://mlsanity.com/paper/1908.00176/) (2019)
- [Multilingual Twitter Corpus and Baselines for Evaluating Demographic Bias in Hate Speech Recognition](https://mlsanity.com/paper/2002.10361/) (2020)
- [Characterizing Intersectional Group Fairness with Worst-Case Comparisons](https://mlsanity.com/paper/2101.01673/) (2021)
- [Fairness and Robustness in Invariant Learning: A Case Study in Toxicity Classification](https://mlsanity.com/paper/2011.06485/) (2020)
- [LiFT: A Scalable Framework for Measuring Fairness in ML Applications](https://mlsanity.com/paper/2008.07433/) (2020)
- [Fairness On The Ground: Applying Algorithmic Fairness Approaches to Production Systems](https://mlsanity.com/paper/2103.06172/) (2021)
- [FAT Forensics: A Python Toolbox for Algorithmic Fairness, Accountability and Transparency](https://mlsanity.com/paper/1909.05167/) (2019)
- [Measuring Disparate Outcomes of Content Recommendation Algorithms with Distributional Inequality Metrics](https://mlsanity.com/paper/2202.01615/) (2022)
- [Individually Fair Ranking](https://mlsanity.com/paper/2103.11023/) (2021)
- [Fairness through Experimentation: Inequality in A/B testing as an approach to responsible design](https://mlsanity.com/paper/2002.05819/) (2020)

## Built on by (most cited first, 20 of 20)

- [Perturbation Augmentation for Fairer NLP](https://mlsanity.com/paper/2205.12586/) (2022)
- [Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML](https://mlsanity.com/paper/2303.08485/) (2023)
- [To the Fairness Frontier and Beyond: Identifying, Quantifying, and Optimizing the Fairness-Accuracy Pareto Frontier](https://mlsanity.com/paper/2206.00074/) (2022)
- [Fair Enough: Standardizing Evaluation and Model Selection for Fairness Research in NLP](https://mlsanity.com/paper/2302.05711/) (2023)
- [Toward Operationalizing Pipeline-aware ML Fairness: A Research Agenda for Developing Practical Guidelines and Tools](https://mlsanity.com/paper/2309.17337/) (2023)
- [Fairness and Sequential Decision Making: Limits, Lessons, and Opportunities](https://mlsanity.com/paper/2301.05753/) (2023)
- [Fairness Hacking: The Malicious Practice of Shrouding Unfairness in Algorithms](https://mlsanity.com/paper/2311.06826/) (2023)
- [The Intersectionality Problem for Algorithmic Fairness](https://mlsanity.com/paper/2411.02569/) (2024)
- [Are fairness metric scores enough to assess discrimination biases in machine learning?](https://mlsanity.com/paper/2306.05307/) (2023)
- [How to Evaluate Automatic Speech Recognition: Comparing Different Performance and Bias Measures](https://mlsanity.com/paper/2507.05885/) (2025)
- [A Keyword Based Approach to Understanding the Overpenalization of Marginalized Groups by English Marginal Abuse Models on Twitter](https://mlsanity.com/paper/2210.06351/) (2022)
- [Data Bias Management](https://mlsanity.com/paper/2305.09686/) (2023)
- [LUCID-GAN: Conditional Generative Models to Locate Unfairness](https://mlsanity.com/paper/2307.15466/) (2023)
- [Measuring and Mitigating Bias for Tabular Datasets with Multiple Protected Attributes](https://mlsanity.com/paper/2405.19300/) (2024)
- [Fairmetrics: An R package for group fairness evaluation](https://mlsanity.com/paper/2506.06243/) (2025)
- [(Un)certainty of (Un)fairness: Preference-Based Selection of Certainly Fair Decision-Makers](https://mlsanity.com/paper/2409.12677/) (2024)
- [A Tutorial On Intersectionality in Fair Rankings](https://mlsanity.com/paper/2502.05333/) (2025)
- [Fairness-in-the-Workflow: How Machine Learning Practitioners at Big Tech Companies Approach Fairness in Recommender Systems](https://mlsanity.com/paper/2505.19441/) (2025)
- [Data Bias Mitigation under Coverage Constraints & The Price of Fairness](https://mlsanity.com/paper/2606.20461/) (2026)
- [FRAME: Separating sampling variation from performance disparities in medical image analysis](https://mlsanity.com/paper/2608.25981/) (2026)

## Abstract (arXiv)

When a model's performance differs across socially or culturally relevant groups--like race, gender, or the intersections of many such groups--it is often called "biased." While much of the work in algorithmic fairness over the last several years has focused on developing various definitions of model fairness (the absence of group-wise model performance disparities) and eliminating such "bias," much less work has gone into rigorously measuring it. In practice, it important to have high quality, human digestible measures of model performance disparities and associated uncertainty quantification about them that can serve as inputs into multi-faceted decision-making processes. In this paper, we show both mathematically and through simulation that many of the metrics used to measure group-wise model performance disparities are themselves statistically biased estimators of the underlying quantities they purport to represent. We argue that this can cause misleading conclusions about the relative group-wise model performance disparities along different dimensions, especially in cases where some sensitive variables consist of categories with few members. We propose the "double-corrected" variance estimator, which provides unbiased estimates and uncertainty quantification of the variance of model performance across groups. It is conceptually simple and easily implementable without statistical software package or numerical optimization. We demonstrate the utility of this approach through simulation and show on a real dataset that while statistically biased estimators of group-wise model performance disparities indicate statistically significant differences, when accounting for statistical bias in the estimator, the estimated between-group disparities are no longer statistically significant.