# SoK: What Makes Private Learning Unfair?

arXiv: https://arxiv.org/abs/2501.14414 · HTML: https://mlsanity.com/paper/2501.14414/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 68 references that are in the map.
- First posted on arXiv 24 Jan 2025.

## Builds on (most cited first, 30 of 36)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [Algorithmic decision making and the cost of fairness](https://mlsanity.com/paper/1701.08230/) (2017)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data](https://mlsanity.com/paper/1610.05755/) (2016)
- [Differential Privacy Has Disparate Impact on Model Accuracy](https://mlsanity.com/paper/1905.12101/) (2019)
- [How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy](https://mlsanity.com/paper/2303.00654/) (2023)
- [Why Is My Classifier Discriminatory?](https://mlsanity.com/paper/1805.12002/) (2018)
- [Hessian-based Analysis of Large Batch Training and Robustness to Adversaries](https://mlsanity.com/paper/1802.08241/) (2018)
- [Differentially Private Fair Learning](https://mlsanity.com/paper/1812.02696/) (2018)
- [Neither Private Nor Fair: Impact of Data Imbalance on Utility and Fairness in Differential Privacy](https://mlsanity.com/paper/2009.06389/) (2020)
- [Differential Privacy and Fairness in Decisions and Learning Tasks: A Survey](https://mlsanity.com/paper/2202.08187/) (2022)
- [On the Convergence and Calibration of Deep Learning with Differential Privacy](https://mlsanity.com/paper/2106.07830/) (2021)
- [DP-SGD vs PATE: Which Has Less Disparate Impact on Model Accuracy?](https://mlsanity.com/paper/2106.12576/) (2021)
- [Disparate Impact in Differential Privacy from Gradient Misalignment](https://mlsanity.com/paper/2206.07737/) (2022)
- [Awareness in Practice: Tensions in Access to Sensitive Attribute Data for Antidiscrimination](https://mlsanity.com/paper/1912.06171/) (2019)
- [DPNAS: Neural Architecture Search for Deep Learning with Differential Privacy](https://mlsanity.com/paper/2110.08557/) (2021)
- [How unfair is private learning ?](https://mlsanity.com/paper/2206.03985/) (2022)
- [Differential Privacy has Bounded Impact on Fairness in Classification](https://mlsanity.com/paper/2210.16242/) (2022)
- [Private, fair and accurate: Training large-scale, privacy-preserving AI models in medical imaging](https://mlsanity.com/paper/2302.01622/) (2023)
- [The Impossibility Theorem of Machine Fairness -- A Causal Perspective](https://mlsanity.com/paper/2007.06024/) (2020)
- [The impossibility of "fairness": a generalized impossibility result for decisions](https://mlsanity.com/paper/1707.01195/) (2017)
- [An Empirical Analysis of Fairness Notions under Differential Privacy](https://mlsanity.com/paper/2302.02910/) (2023)

## Built on by (most cited first, 3 of 3)

- [Where to Intervene? Benchmarking Fairness-Aware Learning on Differentially Private Synthetic Tabular Data](https://mlsanity.com/paper/2607.07471/) (2026)
- [Fair Play for Individuals, Foul Play for Groups? Auditing Anonymization's Impact on ML Fairness](https://mlsanity.com/paper/2505.07985/) (2025)
- [SoK: Colluding Adversaries in Machine Learning Pipelines](https://mlsanity.com/paper/2606.10091/) (2026)

## Abstract (arXiv)

Differential privacy has emerged as the most studied framework for privacy-preserving machine learning. However, recent studies show that enforcing differential privacy guarantees can not only significantly degrade the utility of the model, but also amplify existing disparities in its predictive performance across demographic groups. Although there is extensive research on the identification of factors that contribute to this phenomenon, we still lack a complete understanding of the mechanisms through which differential privacy exacerbates disparities. The literature on this problem is muddled by varying definitions of fairness, differential privacy mechanisms, and inconsistent experimental settings, often leading to seemingly contradictory results. This survey provides the first comprehensive overview of the factors that contribute to the disparate effect of training models with differential privacy guarantees. We discuss their impact and analyze their causal role in such a disparate effect. Our analysis is guided by a taxonomy that categorizes these factors by their position within the machine learning pipeline, allowing us to draw conclusions about their interaction and the feasibility of potential mitigation strategies. We find that factors related to the training dataset and the underlying distribution play a decisive role in the occurrence of disparate impact, highlighting the need for research on these factors to address the issue.