# Fair Enough: Searching for Sufficient Measures of Fairness

arXiv: https://arxiv.org/abs/2110.13029 · HTML: https://mlsanity.com/paper/2110.13029/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 93 references that are in the map.
- First posted on arXiv 25 Oct 2021.

## Builds on (most cited first, 30 of 34)

- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014)
- [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016)
- [On Fairness and Calibration](https://mlsanity.com/paper/1709.02012/) (2017)
- [Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness](https://mlsanity.com/paper/1711.05144/) (2017)
- [AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias](https://mlsanity.com/paper/1810.01943/) (2018)
- [Fairness in Criminal Justice Risk Assessments: The State of the Art](https://mlsanity.com/paper/1703.09207/) (2017)
- [A comparative study of fairness-enhancing interventions in machine learning](https://mlsanity.com/paper/1802.04422/) (2018)
- [Classification with Fairness Constraints: A Meta-Algorithm with Provable Guarantees](https://mlsanity.com/paper/1806.06055/) (2018)
- [Fairness Testing: Testing Software for Discrimination](https://mlsanity.com/paper/1709.03221/) (2017)
- [On the Apparent Conflict Between Individual and Group Fairness](https://mlsanity.com/paper/1912.06883/) (2019)
- [Optimization with Non-Differentiable Constraints with Applications to Fairness, Recall, Churn, and Other Goals](https://mlsanity.com/paper/1809.04198/) (2018)
- [Fair Algorithms for Clustering](https://mlsanity.com/paper/1901.02393/) (2019)
- [Bias in Machine Learning Software: Why? How? What to do?](https://mlsanity.com/paper/2105.12195/) (2021)
- [Ranking with Fairness Constraints](https://mlsanity.com/paper/1704.06840/) (2017)
- [Automated Directed Fairness Testing](https://mlsanity.com/paper/1807.00468/) (2018)
- [Achieving Fairness through Adversarial Learning: an Application to Recidivism Prediction](https://mlsanity.com/paper/1807.00199/) (2018)
- [Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American English](https://mlsanity.com/paper/1707.00061/) (2017)
- [iFair: Learning Individually Fair Data Representations for Algorithmic Decision Making](https://mlsanity.com/paper/1806.01059/) (2018)
- [Do the Machine Learning Models on a Crowd Sourced Platform Exhibit Bias? An Empirical Study on Model Fairness](https://mlsanity.com/paper/2005.12379/) (2020)
- [Residual Unfairness in Fair Machine Learning from Prejudiced Data](https://mlsanity.com/paper/1806.02887/) (2018)
- [Improved Adversarial Learning for Fair Classification](https://mlsanity.com/paper/1901.10443/) (2019)
- [Making Fair ML Software using Trustworthy Explanation](https://mlsanity.com/paper/2007.02893/) (2020)
- [Addressing Fairness in Classification with a Model-Agnostic Multi-Objective Algorithm](https://mlsanity.com/paper/2009.04441/) (2020)
- [Software Engineering for Fairness: A Case Study with Hyperparameter Optimization](https://mlsanity.com/paper/1905.05786/) (2019)
- [Fair Adversarial Gradient Tree Boosting](https://mlsanity.com/paper/1911.05369/) (2019)
- [Evaluating Fairness Metrics in the Presence of Dataset Bias](https://mlsanity.com/paper/1809.09245/) (2018)
- [An Empirical Study on Learning Fairness Metrics for COMPAS Data with Human Supervision](https://mlsanity.com/paper/1910.10255/) (2019)
- [FairPrep: Promoting Data to a First-Class Citizen in Studies on Fairness-Enhancing Interventions](https://mlsanity.com/paper/1911.12587/) (2019)
- [The Impact of Data Preparation on the Fairness of Software Systems](https://mlsanity.com/paper/1910.02321/) (2019)

## Built on by (most cited first, 14 of 14)

- [FRAPPE: A Group Fairness Framework for Post-Processing Everything](https://mlsanity.com/paper/2312.02592/) (2023)
- [Fairness Testing of Large Language Models in Role-Playing](https://mlsanity.com/paper/2411.00585/) (2024)
- [Causality-Aided Trade-off Analysis for Machine Learning Fairness](https://mlsanity.com/paper/2305.13057/) (2023)
- [DCA-Bench: A Benchmark for Dataset Curation Agents](https://mlsanity.com/paper/2406.07275/) (2024)
- [Fair Models in Credit: Intersectional Discrimination and the Amplification of Inequity](https://mlsanity.com/paper/2308.02680/) (2023)
- [Is it Still Fair? A Comparative Evaluation of Fairness Algorithms through the Lens of Covariate Drift](https://mlsanity.com/paper/2409.12428/) (2024)
- [FairSense: Long-Term Fairness Analysis of ML-Enabled Systems](https://mlsanity.com/paper/2501.01665/) (2025)
- [Navigating Fairness: Practitioners' Understanding, Challenges, and Strategies in AI/ML Development](https://mlsanity.com/paper/2403.15481/) (2024)
- [Fairness-aware Multiobjective Evolutionary Learning](https://mlsanity.com/paper/2409.18499/) (2024)
- [From Expectation to Habit: Why Do Software Practitioners Adopt Fairness Toolkits?](https://mlsanity.com/paper/2412.13846/) (2024)
- [Data Preparation for Fairness-Performance Trade-Offs: A Practitioner-Friendly Alternative?](https://mlsanity.com/paper/2412.15920/) (2024)
- [Whence Is A Model Fair? Fixing Fairness Bugs via Propensity Score Matching](https://mlsanity.com/paper/2504.17066/) (2025)
- [FAIRTOPIA: Envisioning Multi-Agent Guardianship for Disrupting Unfair AI Pipelines](https://mlsanity.com/paper/2506.09107/) (2025)
- [Fair Classification by Direct Intervention on Operating Characteristics](https://mlsanity.com/paper/2509.25481/) (2025)

## Abstract (arXiv)

Testing machine learning software for ethical bias has become a pressing current concern. In response, recent research has proposed a plethora of new fairness metrics, for example, the dozens of fairness metrics in the IBM AIF360 toolkit. This raises the question: How can any fairness tool satisfy such a diverse range of goals? While we cannot completely simplify the task of fairness testing, we can certainly reduce the problem. This paper shows that many of those fairness metrics effectively measure the same thing. Based on experiments using seven real-world datasets, we find that (a) 26 classification metrics can be clustered into seven groups, and (b) four dataset metrics can be clustered into three groups. Further, each reduced set may actually predict different things. Hence, it is no longer necessary (or even possible) to satisfy all fairness metrics. In summary, to simplify the fairness testing problem, we recommend the following steps: (1)~determine what type of fairness is desirable (and we offer a handful of such types); then (2) lookup those types in our clusters; then (3) just test for one item per cluster.