# A comparative study of fairness-enhancing interventions in machine learning

arXiv: https://arxiv.org/abs/1802.04422 · HTML: https://mlsanity.com/paper/1802.04422/

## Key facts

- Cited by 208 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Feb 2018.

## Built on by (most cited first, 30 of 208)

- [Trustworthy AI: From Principles to Practices](https://mlsanity.com/paper/2110.01167/) (2021)
- [A survey on datasets for fairness-aware machine learning](https://mlsanity.com/paper/2110.00530/) (2021)
- [Bias Mitigation for Machine Learning Classifiers: A Comprehensive Survey](https://mlsanity.com/paper/2207.07068/) (2022)
- [Counterfactual Explanations Can Be Manipulated](https://mlsanity.com/paper/2106.02666/) (2021)
- [The Fallacy of AI Functionality](https://mlsanity.com/paper/2206.09511/) (2022)
- [Towards Unbiased and Accurate Deferral to Multiple Experts](https://mlsanity.com/paper/2102.13004/) (2021)
- [MEDFAIR: Benchmarking Fairness for Medical Imaging](https://mlsanity.com/paper/2210.01725/) (2022)
- [Benchmarking Distribution Shift in Tabular Data with TableShift](https://mlsanity.com/paper/2312.07577/) (2023)
- [Fairness in Credit Scoring: Assessment, Implementation and Profit Implications](https://mlsanity.com/paper/2103.01907/) (2021)
- [Towards Intersectionality in Machine Learning: Including More Identities, Handling Underrepresentation, and Performing Evaluation](https://mlsanity.com/paper/2205.04610/) (2022)
- [Directional Bias Amplification](https://mlsanity.com/paper/2102.12594/) (2021)
- [Bias Testing and Mitigation in LLM-based Code Generation](https://mlsanity.com/paper/2309.14345/) (2023)
- [Fair Preprocessing: Towards Understanding Compositional Fairness of Data Transformers in Machine Learning Pipeline](https://mlsanity.com/paper/2106.06054/) (2021)
- [Turning the Tables: Biased, Imbalanced, Dynamic Tabular Datasets for ML Evaluation](https://mlsanity.com/paper/2211.13358/) (2022)
- [Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview](https://mlsanity.com/paper/2206.07438/) (2022)
- [Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits](https://mlsanity.com/paper/2205.06922/) (2022)
- [Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2502.06872/) (2025)
- [On the application of Large Language Models for language teaching and assessment technology](https://mlsanity.com/paper/2307.08393/) (2023)
- [On The Impact of Machine Learning Randomness on Group Fairness](https://mlsanity.com/paper/2307.04138/) (2023)
- [Does RAG Introduce Unfairness in LLMs? Evaluating Fairness in Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2409.19804/) (2024)
- ["Garbage In, Garbage Out" Revisited: What Do Machine Learning Application Papers Report About Human-Labeled Training Data?](https://mlsanity.com/paper/2107.02278/) (2021)
- [A Sociotechnical View of Algorithmic Fairness](https://mlsanity.com/paper/2110.09253/) (2021)
- [Minimax Demographic Group Fairness in Federated Learning](https://mlsanity.com/paper/2201.08304/) (2022)
- [When Fair Ranking Meets Uncertain Inference](https://mlsanity.com/paper/2105.02091/) (2021)
- [FairNeuron: Improving Deep Neural Network Fairness with Adversary Games on Selective Neurons](https://mlsanity.com/paper/2204.02567/) (2022)
- [De-biasing "bias" measurement](https://mlsanity.com/paper/2205.05770/) (2022)
- [Enforcing Group Fairness in Algorithmic Decision Making: Utility Maximization Under Sufficiency](https://mlsanity.com/paper/2206.02237/) (2022)
- [Aleatoric and Epistemic Discrimination: Fundamental Limits of Fairness Interventions](https://mlsanity.com/paper/2301.11781/) (2023)
- [A Framework for Fairness: A Systematic Review of Existing Fair AI Solutions](https://mlsanity.com/paper/2112.05700/) (2021)
- [The Possibility of Fairness: Revisiting the Impossibility Theorem in Practice](https://mlsanity.com/paper/2302.06347/) (2023)

## Abstract (arXiv)

Computers are increasingly used to make decisions that have significant impact in people's lives. Often, these predictions can affect different population subgroups disproportionately. As a result, the issue of fairness has received much recent interest, and a number of fairness-enhanced classifiers and predictors have appeared in the literature. This paper seeks to study the following questions: how do these different techniques fundamentally compare to one another, and what accounts for the differences? Specifically, we seek to bring attention to many under-appreciated aspects of such fairness-enhancing interventions. Concretely, we present the results of an open benchmark we have developed that lets us compare a number of different algorithms under a variety of fairness measures, and a large number of existing datasets. We find that although different algorithms tend to prefer specific formulations of fairness preservations, many of these measures strongly correlate with one another. In addition, we find that fairness-preserving algorithms tend to be sensitive to fluctuations in dataset composition (simulated in our benchmark by varying training-test splits), indicating that fairness interventions might be more brittle than previously thought.