# Neither Private Nor Fair: Impact of Data Imbalance on Utility and Fairness in Differential Privacy

arXiv: https://arxiv.org/abs/2009.06389 · HTML: https://mlsanity.com/paper/2009.06389/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Sep 2020.

## Built on by (most cited first, 30 of 61)

- [Robin Hood and Matthew Effects: Differential Privacy Has Disparate Impact on Synthetic Data](https://mlsanity.com/paper/2109.11429/) (2021)
- [Differential Privacy and Fairness in Decisions and Learning Tasks: A Survey](https://mlsanity.com/paper/2202.08187/) (2022)
- [DP-SGD vs PATE: Which Has Less Disparate Impact on Model Accuracy?](https://mlsanity.com/paper/2106.12576/) (2021)
- [Disparate Impact in Differential Privacy from Gradient Misalignment](https://mlsanity.com/paper/2206.07737/) (2022)
- [Privacy Regularization: Joint Privacy-Utility Optimization in Language Models](https://mlsanity.com/paper/2103.07567/) (2021)
- [Interpretable Data-Based Explanations for Fairness Debugging](https://mlsanity.com/paper/2112.09745/) (2021)
- [Privacy and Fairness in Federated Learning: on the Perspective of Trade-off](https://mlsanity.com/paper/2306.14123/) (2023)
- [Differential Privacy has Bounded Impact on Fairness in Classification](https://mlsanity.com/paper/2210.16242/) (2022)
- [Private, fair and accurate: Training large-scale, privacy-preserving AI models in medical imaging](https://mlsanity.com/paper/2302.01622/) (2023)
- [Unlocking Accuracy and Fairness in Differentially Private Image Classification](https://mlsanity.com/paper/2308.10888/) (2023)
- [Demographic-Reliant Algorithmic Fairness: Characterizing the Risks of Demographic Data Collection in the Pursuit of Fairness](https://mlsanity.com/paper/2205.01038/) (2022)
- [The Elusive Pursuit of Reproducing PATE-GAN: Benchmarking, Auditing, Debugging](https://mlsanity.com/paper/2406.13985/) (2024)
- [Style Pooling: Automatic Text Style Obfuscation for Improved Classification Fairness](https://mlsanity.com/paper/2109.04624/) (2021)
- [Recent Advances of Differential Privacy in Centralized Deep Learning: A Systematic Survey](https://mlsanity.com/paper/2309.16398/) (2023)
- [A Fairness Analysis on Private Aggregation of Teacher Ensembles](https://mlsanity.com/paper/2109.08630/) (2021)
- [The Impact of Differential Privacy on Group Disparity Mitigation](https://mlsanity.com/paper/2203.02745/) (2022)
- [SoK: Differential Privacy on Graph-Structured Data](https://mlsanity.com/paper/2203.09205/) (2022)
- [Learning with Impartiality to Walk on the Pareto Frontier of Fairness, Privacy, and Utility](https://mlsanity.com/paper/2302.09183/) (2023)
- [Incentives in Private Collaborative Machine Learning](https://mlsanity.com/paper/2404.01676/) (2024)
- [VisCUIT: Visual Auditor for Bias in CNN Image Classifier](https://mlsanity.com/paper/2204.05899/) (2022)
- [DP-SGD vs PATE: Which Has Less Disparate Impact on GANs?](https://mlsanity.com/paper/2111.13617/) (2021)
- [Preserving privacy in domain transfer of medical AI models comes at no performance costs: The integral role of differential privacy](https://mlsanity.com/paper/2306.06503/) (2023)
- [A Systematic and Formal Study of the Impact of Local Differential Privacy on Fairness: Preliminary Results](https://mlsanity.com/paper/2405.14725/) (2024)
- [Trustworthy Machine Learning via Memorization and the Granular Long-Tail: A Survey on Interactions, Tradeoffs, and Beyond](https://mlsanity.com/paper/2503.07501/) (2025)
- [Protect Your Score: Contact Tracing With Differential Privacy Guarantees](https://mlsanity.com/paper/2312.11581/) (2023)
- [On the Fairness Impacts of Private Ensembles Models](https://mlsanity.com/paper/2305.11807/) (2023)
- [Trade-Offs Between Fairness and Privacy in Language Modeling](https://mlsanity.com/paper/2305.14936/) (2023)
- [Toward the Tradeoffs between Privacy, Fairness and Utility in Federated Learning](https://mlsanity.com/paper/2311.18190/) (2023)
- [On the Impact of Multi-dimensional Local Differential Privacy on Fairness](https://mlsanity.com/paper/2312.04404/) (2023)
- [SoK: Taming the Triangle -- On the Interplays between Fairness, Interpretability and Privacy in Machine Learning](https://mlsanity.com/paper/2312.16191/) (2023)

## Abstract (arXiv)

Deployment of deep learning in different fields and industries is growing day by day due to its performance, which relies on the availability of data and compute. Data is often crowd-sourced and contains sensitive information about its contributors, which leaks into models that are trained on it. To achieve rigorous privacy guarantees, differentially private training mechanisms are used. However, it has recently been shown that differential privacy can exacerbate existing biases in the data and have disparate impacts on the accuracy of different subgroups of data. In this paper, we aim to study these effects within differentially private deep learning. Specifically, we aim to study how different levels of imbalance in the data affect the accuracy and the fairness of the decisions made by the model, given different levels of privacy. We demonstrate that even small imbalances and loose privacy guarantees can cause disparate impacts.