mlsanityData to 14 Oct 2026

Fair Enough: Searching for Sufficient Measures of Fairness

arXiv 2110.13029 ↗First posted 25 Oct 2021cs.LG cs.CY cs.SEcited by 14

Not on the map. Why?

Abstract (arXiv)

Testing machine learning software for ethical bias has become a pressing current concern. In response, recent research has proposed a plethora of new fairness metrics, for example, the dozens of fairness metrics in the IBM AIF360 toolkit. This raises the question: How can any fairness tool satisfy such a diverse range of goals? While we cannot completely simplify the task of fairness testing, we can certainly reduce the problem. This paper shows that many of those fairness metrics effectively measure the same thing. Based on experiments using seven real-world datasets, we find that (a) 26 classification metrics can be clustered into seven groups, and (b) four dataset metrics can be clustered into three groups. Further, each reduced set may actually predict different things. Hence, it is no longer necessary (or even possible) to satisfy all fairness metrics. In summary, to simplify the fairness testing problem, we recommend the following steps: (1)~determine what type of fairness is desirable (and we offer a handful of such types); then (2) lookup those types in our clusters; then (3) just test for one item per cluster.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 34 of its 93 references that are in the map. First posted on arXiv 25 Oct 2021.

Builds on

34 of 93 references.

References · 34

  1. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
  2. 2014Certifying and removing disparate impactcited by 715
  3. 2016Inherent Trade-Offs in the Fair Determination of Risk Scorescited by 586
  4. 2017On Fairness and Calibrationcited by 406
  5. 2017Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairnesscited by 384
  6. 2018AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Biascited by 319
  7. 2017Fairness in Criminal Justice Risk Assessments: The State of the Artcited by 315
  8. 2018A comparative study of fairness-enhancing interventions in machine learningcited by 208
  9. 2018Classification with Fairness Constraints: A Meta-Algorithm with Provable Guaranteescited by 146
  10. 2017Fairness Testing: Testing Software for Discriminationcited by 126
  11. 2019On the Apparent Conflict Between Individual and Group Fairnesscited by 126
  12. 2018Optimization with Non-Differentiable Constraints with Applications to Fairness, Recall, Churn, and Other Goalscited by 103
22 more
  1. 2019Fair Algorithms for Clusteringcited by 98
  2. 2021Bias in Machine Learning Software: Why? How? What to do?cited by 93
  3. 2017Ranking with Fairness Constraintscited by 78
  4. 2018Automated Directed Fairness Testingcited by 78
  5. 2018Achieving Fairness through Adversarial Learning: an Application to Recidivism Predictioncited by 77
  6. 2017Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American Englishcited by 71
  7. 2018iFair: Learning Individually Fair Data Representations for Algorithmic Decision Makingcited by 64
  8. 2020Do the Machine Learning Models on a Crowd Sourced Platform Exhibit Bias? An Empirical Study on Model Fairnesscited by 52
  9. 2018Residual Unfairness in Fair Machine Learning from Prejudiced Datacited by 49
  10. 2019Improved Adversarial Learning for Fair Classificationcited by 25
  11. 2020Making Fair ML Software using Trustworthy Explanationcited by 24
  12. 2020Addressing Fairness in Classification with a Model-Agnostic Multi-Objective Algorithmcited by 23
  13. 2019Software Engineering for Fairness: A Case Study with Hyperparameter Optimizationcited by 18
  14. 2019Fair Adversarial Gradient Tree Boostingcited by 17
  15. 2018Evaluating Fairness Metrics in the Presence of Dataset Biascited by 15
  16. 2019An Empirical Study on Learning Fairness Metrics for COMPAS Data with Human Supervisioncited by 13
  17. 2019FairPrep: Promoting Data to a First-Class Citizen in Studies on Fairness-Enhancing Interventionscited by 13
  18. 2019The Impact of Data Preparation on the Fairness of Software Systemscited by 12
  19. 2020Debiasing classifiers: is reality at variance with expectation?cited by 10
  20. 2020Fairkit, Fairkit, on the Wall, Who's the Fairest of Them All? Supporting Data Scientists in Training Fair Modelscited by 9
  21. 2019Measuring Unfairness through Game-Theoretic Interpretabilitycited by 7
  22. 2020Metrics and methods for a systematic comparison of fairness-aware machine learning algorithmscited by 7

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 7; 2025: 4.

14 papers.

Sort:

By area:

  1. 2023FRAPPE: A Group Fairness Framework for Post-Processing Everythingcited by 16
  2. 2024Fairness Testing of Large Language Models in Role-Playingcited by 11
  3. 2023Causality-Aided Trade-off Analysis for Machine Learning Fairnesscited by 6
  4. 2024DCA-Bench: A Benchmark for Dataset Curation Agentscited by 6
  5. 2023Fair Models in Credit: Intersectional Discrimination and the Amplification of Inequitycited by 3
  6. 2024Is it Still Fair? A Comparative Evaluation of Fairness Algorithms through the Lens of Covariate Driftcited by 3
  7. 2025FairSense: Long-Term Fairness Analysis of ML-Enabled Systemscited by 3
  8. 2024Navigating Fairness: Practitioners' Understanding, Challenges, and Strategies in AI/ML Developmentcited by 1
  9. 2024Fairness-aware Multiobjective Evolutionary Learningcited by 1
  10. 2024From Expectation to Habit: Why Do Software Practitioners Adopt Fairness Toolkits?cited by 1
  11. 2024Data Preparation for Fairness-Performance Trade-Offs: A Practitioner-Friendly Alternative?cited by 0
  12. 2025Whence Is A Model Fair? Fixing Fairness Bugs via Propensity Score Matchingcited by 0
2 more
  1. 2025FAIRTOPIA: Envisioning Multi-Agent Guardianship for Disrupting Unfair AI Pipelinescited by 0
  2. 2025Fair Classification by Direct Intervention on Operating Characteristicscited by 0