# Failing on Bias Mitigation: A Case Study on the Challenges of Fairness in Government Data

arXiv: https://arxiv.org/abs/2601.17054 · HTML: https://mlsanity.com/paper/2601.17054/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 45 references that are in the map.
- First posted on arXiv 21 Jan 2026.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [On Fairness and Calibration](https://mlsanity.com/paper/1709.02012/) (2017)
- [Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment](https://mlsanity.com/paper/1610.08452/) (2016)
- [Delayed Impact of Fair Machine Learning](https://mlsanity.com/paper/1803.04383/) (2018)
- [Fair Regression: Quantitative Definitions and Reduction-based Algorithms](https://mlsanity.com/paper/1905.12843/) (2019)
- [A Convex Framework for Fair Regression](https://mlsanity.com/paper/1706.02409/) (2017)
- [Selective Regression Under Fairness Criteria](https://mlsanity.com/paper/2110.15403/) (2021)
- [Assessing Fairness in the Presence of Missing Data](https://mlsanity.com/paper/2112.04899/) (2021)
- [Supervised Algorithmic Fairness in Distribution Shifts: A Survey](https://mlsanity.com/paper/2402.01327/) (2024)
- [From the Fair Distribution of Predictions to the Fair Distribution of Social Goods: Evaluating the Impact of Fair Machine Learning on Long-Term Unemployment](https://mlsanity.com/paper/2401.14438/) (2024)
- [ProxiMix: Enhancing Fairness with Proximity Samples in Subgroups](https://mlsanity.com/paper/2410.01145/) (2024)

## Abstract (arXiv)

The potential for bias and unfairness in AI-supporting government services raises ethical and legal concerns. Using crime rate prediction with the Bristol City Council data as a case study, we examine how these issues persist. Rather than auditing real-world deployed systems, our goal is to understand why widely adopted bias mitigation techniques often fail when applied to government data. Our findings reveal that bias mitigation approaches applied to government data are not always effective -- not because of flaws in model architecture or metric selection, but due to the inherent properties of the data itself. Through comparing a set of comprehensive models and fairness methods, our experiments consistently show that the mitigation efforts cannot overcome the embedded unfairness in the data -- further reinforcing that the origin of bias lies in the structure and history of government datasets. We then explore the reasons for the mitigation failures in predictive models on government data and highlight the potential sources of unfairness posed by data distribution shifts, the accumulation of historical bias, and delays in data release. We also discover the limitations of the blind spots in fairness analysis and bias mitigation methods when only targeting a single sensitive feature through a set of intersectional fairness experiments. Although this study is limited to one city, the findings are highly suggestive, which can contribute to an early warning that biases in government data may persist even with standard mitigation methods.