mlsanityData to 7 Oct 2026

Robin Hood and Matthew Effects: Differential Privacy Has Disparate Impact on Synthetic Data

arXiv 2109.11429 ↗First posted 23 Sep 2021cs.LG cs.AI cs.CRcited by 48

Not on the map. Why?

Abstract (arXiv)

Generative models trained with Differential Privacy (DP) can be used to generate synthetic data while minimizing privacy risks. We analyze the impact of DP on these models vis-a-vis underrepresented classes/subgroups of data, specifically, studying: 1) the size of classes/subgroups in the synthetic data, and 2) the accuracy of classification tasks run on them. We also evaluate the effect of various levels of imbalance and privacy budgets. Our analysis uses three state-of-the-art DP models (PrivBayes, DP-WGAN, and PATE-GAN) and shows that DP yields opposite size distributions in the generated synthetic data. It affects the gap between the majority and minority classes/subgroups; in some cases by reducing it (a "Robin Hood" effect) and, in others, by increasing it (a "Matthew" effect). Either way, this leads to (similar) disparate impacts on the accuracy of classification tasks on the synthetic data, affecting disproportionately more the underrepresented subparts of the data. Consequently, when training models on synthetic data, one might incur the risk of treating different subpopulations unevenly, leading to unreliable or unfair conclusions.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Builds on 25 of its 65 references that are in the map. First posted on arXiv 23 Sep 2021.

Builds on

25 of 65 references.

References · 25

  1. 2016Deep Learning with Differential Privacycited by 2,682
  2. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  3. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  4. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  5. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  6. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 347
  7. 2019Differential Privacy Has Disparate Impact on Model Accuracycited by 316
  8. 2018Scalable Private Learning with PATEcited by 266
  9. 2018Differentially Private Generative Adversarial Networkcited by 235
  10. 2020Understanding Gradient Clipping in Private SGD: A Geometric Perspectivecited by 140
  11. 2021Winning the NIST Contest: A scalable and general approach to differentially private synthetic datacited by 101
  12. 2021ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Modelscited by 79
13 more
  1. 2018Differentially Private Fair Learningcited by 76
  2. 2020Neither Private Nor Fair: Impact of Data Imbalance on Utility and Fairness in Differential Privacycited by 61
  3. 2019Fair Decision Making using Privacy-Protected Datacited by 58
  4. 2020Differentially Private and Fair Deep Learning: A Lagrangian Dual Approachcited by 53
  5. 2019Diffprivlib: The IBM Differential Privacy Librarycited by 51
  6. 2019Detecting Overfitting of Deep Generative Networks via Latent Recoverycited by 44
  7. 2017Differentially Private Mixture of Generative Neural Networkscited by 39
  8. 2020Chasing Your Long Tails: Differentially Private Prediction in Health Care Settingscited by 37
  9. 2021DP-SGD vs PATE: Which Has Less Disparate Impact on Model Accuracy?cited by 32
  10. 2019Differentially Private Generative Adversarial Networks for Time Series, Continuous, and Discrete Open Datacited by 31
  11. 2019Differentially Private Synthetic Mixed-Type Data Generation For Unsupervised Learningcited by 11
  12. 2021A Fairness Analysis on Private Aggregation of Teacher Ensemblescited by 11
  13. 2021An Analysis of the Deployment of Models Trained on Private Tabular Synthetic Data: Unexpected Surprisescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 6; 2023: 21; 2024: 11; 2025: 4; 2026: 5.

48 papers.

Sort:

By area:

  1. 2023Machine Learning for Synthetic Data Generation: A Reviewcited by 118
  2. 2022Synthetic Data -- what, why and how?cited by 66
  3. 2024Fairness Feedback Loops: Training on Synthetic Data Amplifies Biascited by 55
  4. 2023SoK: Privacy-Preserving Data Synthesiscited by 37
  5. 2023Privacy and Fairness in Federated Learning: on the Perspective of Trade-offcited by 26
  6. 2023A Linear Reconstruction Approach for Attribute Inference Attacks against Synthetic Datacited by 23
  7. 2023Reimagining Synthetic Tabular Data Generation through Data-Centric AI: A Comprehensive Benchmarkcited by 23
  8. 2023Arbitrary Decisions are a Hidden Cost of Differentially Private Trainingcited by 19
  9. 2022Differential Privacy has Bounded Impact on Fairness in Classificationcited by 18
  10. 2024Systematic Assessment of Tabular Data Synthesiscited by 17
  11. 2023The Inadequacy of Similarity-based Privacy Metrics: Privacy Attacks against "Truly Anonymous" Synthetic Datasetscited by 14
  12. 2024Conformal Prediction Sets Can Cause Disparate Impactcited by 14
36 more
  1. 2023Graphical vs. Deep Generative Models: Measuring the Impact of Differentially Private Mechanisms and Budgets on Utilitycited by 13
  2. 2023CuTS: Customizable Tabular Synthetic Data Generationcited by 12
  3. 2024The Elusive Pursuit of Reproducing PATE-GAN: Benchmarking, Auditing, Debuggingcited by 12
  4. 2025Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectivescited by 10
  5. 2023An Empirical Analysis of Fairness Notions under Differential Privacycited by 9
  6. 2023Post-processing Private Synthetic Data for Improving Utility on Selected Measurescited by 9
  7. 2022Differentially Private Shapley Values for Data Evaluationcited by 8
  8. 2023Assessment of Differentially Private Synthetic Data for Utility and Fairness in End-to-End Machine Learning Pipelines for Tabular Datacited by 8
  9. 2023Toward Operationalizing Pipeline-aware ML Fairness: A Research Agenda for Developing Practical Guidelines and Toolscited by 7
  10. 2021DP-SGD vs PATE: Which Has Less Disparate Impact on GANs?cited by 6
  11. 2024Tabular Data Synthesis with Differential Privacy: A Surveycited by 6
  12. 2023(Local) Differential Privacy has NO Disparate Impact on Fairnesscited by 5
  13. 2022Privacy in Practice: Private COVID-19 Detection in X-Ray Images (Extended Version)cited by 4
  14. 2023A Unified View of Differentially Private Deep Generative Modelingcited by 4
  15. 2023Toward the Tradeoffs between Privacy, Fairness and Utility in Federated Learningcited by 4
  16. 2023On the Impact of Multi-dimensional Local Differential Privacy on Fairnesscited by 4
  17. 2023A Simple and Practical Method for Reducing the Disparate Impact of Differential Privacycited by 4
  18. 2023When Synthetic Data Met Regulationcited by 3
  19. 2024PFGuard: A Generative Framework with Privacy and Fairness Safeguardscited by 3
  20. 2022Spending Privacy Budget Fairly and Wiselycited by 2
  21. 2022I Prefer not to Say: Protecting User Consent in Models with Optional Personal Datacited by 1
  22. 2023On the Challenges of Deploying Privacy-Preserving Synthetic Data in the Enterprisecited by 1
  23. 2023A Cautionary Tale: On the Role of Reference Data in Empirical Privacy Defensescited by 1
  24. 2024Privacy Re-identification Attacks on Tabular GANscited by 1
  25. 2024Fairness-Optimized Synthetic EHR Generation for Arbitrary Downstream Predictive Taskscited by 1
  26. 2024The Impact of Generalization Techniques on the Interplay Among Privacy, Utility, and Fairness in Image Classificationcited by 1
  27. 2025Quantitative Auditing of AI Fairness with Differentially Private Synthetic Datacited by 1
  28. 2026Rethinking Anonymity Claims in Synthetic Data Generation: A Model-Centric Privacy Attack Perspectivecited by 1
  29. 2024Almost linear time differentially private release of synthetic graphscited by 0
  30. 2024SAFES: Sequential Privacy and Fairness Enhancing Data Synthesis for Responsible AIcited by 0
  31. 2025FedPF: Accurate Target Privacy Preserving Federated Learning Balancing Fairness and Utilitycited by 0
  32. 2025Optimal Fairness under Local Differential Privacycited by 0
  33. 2026Differential Privacy in Two-Layer Networks: How DP-SGD Harms Fairness and Robustnesscited by 0
  34. 2026Decoupling Identity from Utility: Privacy-by-Design Frameworks for Financial Ecosystemscited by 0
  35. 2026Tight Auditing of Differential Privacy in MST and AIMcited by 0
  36. 2026Disparate Impact in Synthetic Data Generationcited by 0