# Robin Hood and Matthew Effects: Differential Privacy Has Disparate Impact on Synthetic Data

arXiv: https://arxiv.org/abs/2109.11429 · HTML: https://mlsanity.com/paper/2109.11429/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 65 references that are in the map.
- First posted on arXiv 23 Sep 2021.

## Builds on (most cited first, 25 of 25)

- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Does Learning Require Memorization? A Short Tale about a Long Tail](https://mlsanity.com/paper/1906.05271/) (2019)
- [Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data](https://mlsanity.com/paper/1610.05755/) (2016)
- [Differential Privacy Has Disparate Impact on Model Accuracy](https://mlsanity.com/paper/1905.12101/) (2019)
- [Scalable Private Learning with PATE](https://mlsanity.com/paper/1802.08908/) (2018)
- [Differentially Private Generative Adversarial Network](https://mlsanity.com/paper/1802.06739/) (2018)
- [Understanding Gradient Clipping in Private SGD: A Geometric Perspective](https://mlsanity.com/paper/2006.15429/) (2020)
- [Winning the NIST Contest: A scalable and general approach to differentially private synthetic data](https://mlsanity.com/paper/2108.04978/) (2021)
- [ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Models](https://mlsanity.com/paper/2102.02551/) (2021)
- [Differentially Private Fair Learning](https://mlsanity.com/paper/1812.02696/) (2018)
- [Neither Private Nor Fair: Impact of Data Imbalance on Utility and Fairness in Differential Privacy](https://mlsanity.com/paper/2009.06389/) (2020)
- [Fair Decision Making using Privacy-Protected Data](https://mlsanity.com/paper/1905.12744/) (2019)
- [Diffprivlib: The IBM Differential Privacy Library](https://mlsanity.com/paper/1907.02444/) (2019)
- [Differentially Private and Fair Deep Learning: A Lagrangian Dual Approach](https://mlsanity.com/paper/2009.12562/) (2020)
- [Detecting Overfitting of Deep Generative Networks via Latent Recovery](https://mlsanity.com/paper/1901.03396/) (2019)
- [Differentially Private Mixture of Generative Neural Networks](https://mlsanity.com/paper/1709.04514/) (2017)
- [Chasing Your Long Tails: Differentially Private Prediction in Health Care Settings](https://mlsanity.com/paper/2010.06667/) (2020)
- [DP-SGD vs PATE: Which Has Less Disparate Impact on Model Accuracy?](https://mlsanity.com/paper/2106.12576/) (2021)
- [Differentially Private Generative Adversarial Networks for Time Series, Continuous, and Discrete Open Data](https://mlsanity.com/paper/1901.02477/) (2019)
- [Differentially Private Synthetic Mixed-Type Data Generation For Unsupervised Learning](https://mlsanity.com/paper/1912.03250/) (2019)
- [A Fairness Analysis on Private Aggregation of Teacher Ensembles](https://mlsanity.com/paper/2109.08630/) (2021)
- [An Analysis of the Deployment of Models Trained on Private Tabular Synthetic Data: Unexpected Surprises](https://mlsanity.com/paper/2106.10241/) (2021)

## Built on by (most cited first, 30 of 48)

- [Machine Learning for Synthetic Data Generation: A Review](https://mlsanity.com/paper/2302.04062/) (2023)
- [Synthetic Data -- what, why and how?](https://mlsanity.com/paper/2205.03257/) (2022)
- [Fairness Feedback Loops: Training on Synthetic Data Amplifies Bias](https://mlsanity.com/paper/2403.07857/) (2024)
- [SoK: Privacy-Preserving Data Synthesis](https://mlsanity.com/paper/2307.02106/) (2023)
- [Privacy and Fairness in Federated Learning: on the Perspective of Trade-off](https://mlsanity.com/paper/2306.14123/) (2023)
- [A Linear Reconstruction Approach for Attribute Inference Attacks against Synthetic Data](https://mlsanity.com/paper/2301.10053/) (2023)
- [Reimagining Synthetic Tabular Data Generation through Data-Centric AI: A Comprehensive Benchmark](https://mlsanity.com/paper/2310.16981/) (2023)
- [Arbitrary Decisions are a Hidden Cost of Differentially Private Training](https://mlsanity.com/paper/2302.14517/) (2023)
- [Differential Privacy has Bounded Impact on Fairness in Classification](https://mlsanity.com/paper/2210.16242/) (2022)
- [Systematic Assessment of Tabular Data Synthesis](https://mlsanity.com/paper/2402.06806/) (2024)
- [The Inadequacy of Similarity-based Privacy Metrics: Privacy Attacks against "Truly Anonymous" Synthetic Datasets](https://mlsanity.com/paper/2312.05114/) (2023)
- [Conformal Prediction Sets Can Cause Disparate Impact](https://mlsanity.com/paper/2410.01888/) (2024)
- [Graphical vs. Deep Generative Models: Measuring the Impact of Differentially Private Mechanisms and Budgets on Utility](https://mlsanity.com/paper/2305.10994/) (2023)
- [CuTS: Customizable Tabular Synthetic Data Generation](https://mlsanity.com/paper/2307.03577/) (2023)
- [The Elusive Pursuit of Reproducing PATE-GAN: Benchmarking, Auditing, Debugging](https://mlsanity.com/paper/2406.13985/) (2024)
- [Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectives](https://mlsanity.com/paper/2508.02853/) (2025)
- [An Empirical Analysis of Fairness Notions under Differential Privacy](https://mlsanity.com/paper/2302.02910/) (2023)
- [Post-processing Private Synthetic Data for Improving Utility on Selected Measures](https://mlsanity.com/paper/2305.15538/) (2023)
- [Differentially Private Shapley Values for Data Evaluation](https://mlsanity.com/paper/2206.00511/) (2022)
- [Assessment of Differentially Private Synthetic Data for Utility and Fairness in End-to-End Machine Learning Pipelines for Tabular Data](https://mlsanity.com/paper/2310.19250/) (2023)
- [DP-SGD vs PATE: Which Has Less Disparate Impact on GANs?](https://mlsanity.com/paper/2111.13617/) (2021)
- [Toward Operationalizing Pipeline-aware ML Fairness: A Research Agenda for Developing Practical Guidelines and Tools](https://mlsanity.com/paper/2309.17337/) (2023)
- [Tabular Data Synthesis with Differential Privacy: A Survey](https://mlsanity.com/paper/2411.03351/) (2024)
- [(Local) Differential Privacy has NO Disparate Impact on Fairness](https://mlsanity.com/paper/2304.12845/) (2023)
- [Privacy in Practice: Private COVID-19 Detection in X-Ray Images (Extended Version)](https://mlsanity.com/paper/2211.11434/) (2022)
- [A Unified View of Differentially Private Deep Generative Modeling](https://mlsanity.com/paper/2309.15696/) (2023)
- [Toward the Tradeoffs between Privacy, Fairness and Utility in Federated Learning](https://mlsanity.com/paper/2311.18190/) (2023)
- [On the Impact of Multi-dimensional Local Differential Privacy on Fairness](https://mlsanity.com/paper/2312.04404/) (2023)
- [A Simple and Practical Method for Reducing the Disparate Impact of Differential Privacy](https://mlsanity.com/paper/2312.11712/) (2023)
- [When Synthetic Data Met Regulation](https://mlsanity.com/paper/2307.00359/) (2023)

## Abstract (arXiv)

Generative models trained with Differential Privacy (DP) can be used to generate synthetic data while minimizing privacy risks. We analyze the impact of DP on these models vis-a-vis underrepresented classes/subgroups of data, specifically, studying: 1) the size of classes/subgroups in the synthetic data, and 2) the accuracy of classification tasks run on them. We also evaluate the effect of various levels of imbalance and privacy budgets. Our analysis uses three state-of-the-art DP models (PrivBayes, DP-WGAN, and PATE-GAN) and shows that DP yields opposite size distributions in the generated synthetic data. It affects the gap between the majority and minority classes/subgroups; in some cases by reducing it (a "Robin Hood" effect) and, in others, by increasing it (a "Matthew" effect). Either way, this leads to (similar) disparate impacts on the accuracy of classification tasks on the synthetic data, affecting disproportionately more the underrepresented subparts of the data. Consequently, when training models on synthetic data, one might incur the risk of treating different subpopulations unevenly, leading to unreliable or unfair conclusions.