# A Comprehensive Empirical Study of Bias Mitigation Methods for Machine Learning Classifiers

arXiv: https://arxiv.org/abs/2207.03277 · HTML: https://mlsanity.com/paper/2207.03277/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 78 references that are in the map.
- First posted on arXiv 7 Jul 2022.

## Builds on (most cited first, 23 of 23)

- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Mitigating Unwanted Biases with Adversarial Learning](https://mlsanity.com/paper/1801.07593/) (2018)
- [On Fairness and Calibration](https://mlsanity.com/paper/1709.02012/) (2017)
- [Fairness in Criminal Justice Risk Assessments: The State of the Art](https://mlsanity.com/paper/1703.09207/) (2017)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [Classification with Fairness Constraints: A Meta-Algorithm with Provable Guarantees](https://mlsanity.com/paper/1806.06055/) (2018)
- [Fairness Testing: Testing Software for Discrimination](https://mlsanity.com/paper/1709.03221/) (2017)
- [A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual & Group Unfairness via Inequality Indices](https://mlsanity.com/paper/1807.00787/) (2018)
- [Bias in Machine Learning Software: Why? How? What to do?](https://mlsanity.com/paper/2105.12195/) (2021)
- [Automated Directed Fairness Testing](https://mlsanity.com/paper/1807.00468/) (2018)
- [Do the Machine Learning Models on a Crowd Sourced Platform Exhibit Bias? An Empirical Study on Model Fairness](https://mlsanity.com/paper/2005.12379/) (2020)
- [Fair Preprocessing: Towards Understanding Compositional Fairness of Data Transformers in Machine Learning Pipeline](https://mlsanity.com/paper/2106.06054/) (2021)
- [NeuronFair: Interpretable White-Box Fairness Testing through Biased Neuron Identification](https://mlsanity.com/paper/2112.13214/) (2021)
- [Making Fair ML Software using Trustworthy Explanation](https://mlsanity.com/paper/2007.02893/) (2020)
- [A Benchmark Study on Sentiment Analysis for Software Engineering Research](https://mlsanity.com/paper/1803.06525/) (2018)
- [Assessing Software Defection Prediction Performance: Why Using the Matthews Correlation Coefficient Matters](https://mlsanity.com/paper/2003.01182/) (2020)
- [Can We Use SE-specific Sentiment Analysis Tools in a Cross-Platform Setting?](https://mlsanity.com/paper/2004.00300/) (2020)
- [SEntiMoji: An Emoji-Powered Learning Approach for Sentiment Analysis in Software Engineering](https://mlsanity.com/paper/1907.02202/) (2019)
- [A Method to Assess and Argue for Practical Significance in Software Engineering](https://mlsanity.com/paper/1809.09849/) (2018)

## Built on by (most cited first, 30 of 30)

- [Bias Mitigation for Machine Learning Classifiers: A Comprehensive Survey](https://mlsanity.com/paper/2207.07068/) (2022)
- [Bias Testing and Mitigation in LLM-based Code Generation](https://mlsanity.com/paper/2309.14345/) (2023)
- [Fairness Testing of Large Language Models in Role-Playing](https://mlsanity.com/paper/2411.00585/) (2024)
- [Bias Behind the Wheel: Fairness Testing of Autonomous Driving Systems](https://mlsanity.com/paper/2308.02935/) (2023)
- [Causality-Aided Trade-off Analysis for Machine Learning Fairness](https://mlsanity.com/paper/2305.13057/) (2023)
- [Unveiling and Mitigating Bias in Large Language Model Recommendations: A Path to Fairness](https://mlsanity.com/paper/2409.10825/) (2024)
- [M3Fair: Mitigating Bias in Healthcare Data through Multi-Level and Multi-Sensitive-Attribute Reweighting Method](https://mlsanity.com/paper/2306.04118/) (2023)
- [FairCompass: Operationalising Fairness in Machine Learning](https://mlsanity.com/paper/2312.16726/) (2023)
- [Analyzing and Mitigating Bias for Vulnerable Classes: Towards Balanced Representation in Dataset](https://mlsanity.com/paper/2401.10397/) (2024)
- [Diversity Drives Fairness: Ensemble of Higher Order Mutants for Intersectional Fairness of Machine Learning Software](https://mlsanity.com/paper/2412.08167/) (2024)
- [Subgroups Matter for Robust Bias Mitigation](https://mlsanity.com/paper/2505.21363/) (2025)
- [Towards Better Fairness-Utility Trade-off: A Comprehensive Measurement-Based Reinforcement Learning Framework](https://mlsanity.com/paper/2307.11379/) (2023)
- [Reranking individuals: The effect of fair classification within-groups](https://mlsanity.com/paper/2401.13391/) (2024)
- [Software Fairness Dilemma: Is Bias Mitigation a Zero-Sum Game?](https://mlsanity.com/paper/2508.03323/) (2025)
- [Mitigating Prompt-Induced Cognitive Biases in General-Purpose AI for Software Engineering](https://mlsanity.com/paper/2604.16756/) (2026)
- [Fairness Improvement with Multiple Protected Attributes: How Far Are We?](https://mlsanity.com/paper/2308.01923/) (2023)
- [Democratize with Care: The need for fairness specific features in user-interface based open source AutoML tools](https://mlsanity.com/paper/2312.12460/) (2023)
- [Equity in Healthcare: Analyzing Disparities in Machine Learning Predictions of Diabetic Patient Readmissions](https://mlsanity.com/paper/2403.19057/) (2024)
- [Contexts Matter: An Empirical Study on Contextual Influence in Fairness Testing for Deep Learning Systems](https://mlsanity.com/paper/2408.06102/) (2024)
- [Revisiting Technical Bias Mitigation Strategies](https://mlsanity.com/paper/2410.17433/) (2024)
- [Balancing the Scales: Enhancing Fairness in Facial Expression Recognition with Latent Alignment](https://mlsanity.com/paper/2410.19444/) (2024)
- [BiasGuard: Guardrailing Fairness in Machine Learning Production Systems](https://mlsanity.com/paper/2501.04142/) (2025)
- [seeBias: A Comprehensive Tool for Assessing and Visualizing AI Fairness](https://mlsanity.com/paper/2504.08418/) (2025)
- [Whence Is A Model Fair? Fixing Fairness Bugs via Propensity Score Matching](https://mlsanity.com/paper/2504.17066/) (2025)
- [What Must a Fairness Audit Report When Demographic Data Is Incomplete?](https://mlsanity.com/paper/2506.23033/) (2025)
- [Fairness Is Not Just Ethical: Performance Trade-Off via Data Correlation Tuning to Mitigate Bias in ML Software](https://mlsanity.com/paper/2512.21348/) (2025)
- [Should I use Synthetic Data for That? An Analysis of the Suitability of Synthetic Data for Data Sharing and Augmentation](https://mlsanity.com/paper/2602.03791/) (2026)
- [Can Generative Artificial Intelligence Survive Data Contamination? Theoretical Guarantees under Contaminated Recursive Training](https://mlsanity.com/paper/2602.16065/) (2026)
- [FairSelect: A Systematic Evaluation of Multi-Level and Intersectional Algorithmic Fairness](https://mlsanity.com/paper/2607.08953/) (2026)
- [When Post-Processing Fairness Constraints Help and When They Harm: Evidence from Eight Cross-Domain Evaluations](https://mlsanity.com/paper/2609.26955/) (2026)

## Abstract (arXiv)

Software bias is an increasingly important operational concern for software engineers. We present a large-scale, comprehensive empirical study of 17 representative bias mitigation methods for Machine Learning (ML) classifiers, evaluated with 11 ML performance metrics (e.g., accuracy), 4 fairness metrics, and 20 types of fairness-performance trade-off assessment, applied to 8 widely-adopted software decision tasks. The empirical coverage is much more comprehensive, covering the largest numbers of bias mitigation methods, evaluation metrics, and fairness-performance trade-off measures compared to previous work on this important software property. We find that (1) the bias mitigation methods significantly decrease ML performance in 53% of the studied scenarios (ranging between 42%~66% according to different ML performance metrics); (2) the bias mitigation methods significantly improve fairness measured by the 4 used metrics in 46% of all the scenarios (ranging between 24%~59% according to different fairness metrics); (3) the bias mitigation methods even lead to decrease in both fairness and ML performance in 25% of the scenarios; (4) the effectiveness of the bias mitigation methods depends on tasks, models, the choice of protected attributes, and the set of metrics used to assess fairness and ML performance; (5) there is no bias mitigation method that can achieve the best trade-off in all the scenarios. The best method that we find outperforms other methods in 30% of the scenarios. Researchers and practitioners need to choose the bias mitigation method best suited to their intended application scenario(s).