mlsanityData to 7 Oct 2026

The Elusive Pursuit of Reproducing PATE-GAN: Benchmarking, Auditing, Debugging

arXiv 2406.13985 ↗First posted 20 Jun 2024cs.LG cs.CRcited by 12

Not on the map. Why?

Abstract (arXiv)

Synthetic data created by differentially private (DP) generative models is increasingly used in real-world settings. In this context, PATE-GAN has emerged as one of the most popular algorithms, combining Generative Adversarial Networks (GANs) with the private training approach of PATE (Private Aggregation of Teacher Ensembles). In this paper, we set out to reproduce the utility evaluation from the original PATE-GAN paper, compare available implementations, and conduct a privacy audit. More precisely, we analyze and benchmark six open-source PATE-GAN implementations, including three by (a subset of) the original authors. First, we shed light on architecture deviations and empirically demonstrate that none reproduce the utility performance reported in the original paper. We then present an in-depth privacy evaluation, which includes DP auditing, and show that all implementations leak more privacy than intended. Furthermore, we uncover 19 privacy violations and 5 other bugs in these six open-source implementations. Lastly, our codebase is available from: https://github.com/spalabucr/pategan-audit.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 53 of its 94 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

53 of 94 references.

References · 53

  1. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  2. 2016Deep Learning with Differential Privacycited by 2,682
  3. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  4. 2019Modeling Tabular data using Conditional GANcited by 695
  5. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 347
  6. 2017Real-valued (Medical) Time Series Generation with Recurrent Conditional GANscited by 329
  7. 2019Differential Privacy Has Disparate Impact on Model Accuracycited by 316
  8. 2018Scalable Private Learning with PATEcited by 266
  9. 2018Differentially Private Generative Adversarial Networkcited by 235
  10. 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 213
  11. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 193
  12. 2019Reconstruction and Membership Inference Attacks against Generative Modelscited by 122
38 more
  1. 2023Privacy Auditing with One (1) Training Runcited by 119
  2. 2021Winning the NIST Contest: A scalable and general approach to differentially private synthetic datacited by 101
  3. 2022AIM: An Adaptive and Iterative Mechanism for Differentially Private Synthetic Datacited by 88
  4. 2021Benchmarking Differentially Private Synthetic Data Generation Algorithmscited by 80
  5. 2023Membership Inference Attacks against Synthetic Data through Overfitting Detectioncited by 66
  6. 2020Neither Private Nor Fair: Impact of Data Imbalance on Utility and Fairness in Differential Privacycited by 61
  7. 2021Iterative Methods for Private Synthetic Data: Unifying Framework and New Methodscited by 58
  8. 2022TAPAS: a Toolbox for Adversarial Privacy Auditing of Synthetic Datacited by 56
  9. 2019G-PATE: Scalable Differentially Private Data Generator via Private Aggregation of Teacher Discriminatorscited by 53
  10. 2021Differentially Private Query Release Through Adaptive Projectioncited by 50
  11. 2021Robin Hood and Matthew Effects: Differential Privacy Has Disparate Impact on Synthetic Datacited by 48
  12. 2020New Oracle-Efficient Algorithms for Private Synthetic Data Releasecited by 43
  13. 2022Geographic Spines in the 2020 Census Disclosure Avoidance Systemcited by 42
  14. 2017Differentially Private Mixture of Generative Neural Networkscited by 39
  15. 2022Debugging Differential Privacy: A Case Study for Privacy Auditingcited by 39
  16. 2020Differentially Private Synthetic Data: Applied Evaluations and Enhancementscited by 36
  17. 2022Private Synthetic Data for Multitask Learning and Marginal Queriescited by 35
  18. 2023One-shot Empirical Privacy Estimation for Federated Learningcited by 35
  19. 2021DP-SGD vs PATE: Which Has Less Disparate Impact on Model Accuracy?cited by 32
  20. 2019Differentially Private Generative Adversarial Networks for Time Series, Continuous, and Discrete Open Datacited by 31
  21. 2022Bayesian Estimation of Differential Privacycited by 31
  22. 2023Achilles' Heels: Vulnerable Record Identification in Synthetic Data Publishingcited by 28
  23. 2022CANIFE: Crafting Canaries for Empirical Privacy Measurement in Federated Learningcited by 24
  24. 2020Kamino: Constraint-Aware Differentially Private Data Synthesiscited by 23
  25. 2023A Linear Reconstruction Approach for Attribute Inference Attacks against Synthetic Datacited by 23
  26. 2024"What do you want from theory alone?" Experimenting with Tight Auditing of Differentially Private Synthetic Data Generationcited by 20
  27. 2024Nearly Tight Black-Box Auditing of Differentially Private Machine Learningcited by 19
  28. 2021"I need a better description'': An Investigation Into User Expectations For Differential Privacycited by 17
  29. 2024Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Modelcited by 17
  30. 2024Differentially Private Release of Israel's National Registry of Live Birthscited by 16
  31. 2023Graphical vs. Deep Generative Models: Measuring the Impact of Differentially Private Mechanisms and Budgets on Utilitycited by 13
  32. 2022dpart: Differentially Private Autoregressive Tabular, a General Framework for Synthetic Data Generationcited by 12
  33. 2023CuTS: Customizable Tabular Synthetic Data Generationcited by 12
  34. 2023DP-Auditorium: a Large Scale Library for Auditing Differential Privacycited by 11
  35. 2021DP-SGD vs PATE: Which Has Less Disparate Impact on GANs?cited by 6
  36. 2023FLAIM: AIM-based Synthetic Data Generation in the Federated Settingcited by 6
  37. 2023Generating tabular datasets under differential privacycited by 5
  38. 2024Reproducibility Study on Adversarial Attacks Against Robust Transformer Trackerscited by 3

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 7; 2026: 3.

12 papers.

  1. 2025DPImageBench: A Unified Benchmark for Differentially Private Image Synthesiscited by 11
  2. 2024Privacy Vulnerabilities in Marginals-based Synthetic Datacited by 10
  3. 2025Generating Synthetic Data with Formal Privacy Guarantees: State of the Art and the Road Aheadcited by 4
  4. 2025The Importance of Being Discrete: Measuring the Impact of Discretization in End-to-End Differentially Private Synthetic Datacited by 4
  5. 2025The Hitchhiker's Guide to Efficient, End-to-End, and Tight DP Auditingcited by 4
  6. 2025SynBench: A Benchmark for Differentially Private Text Generationcited by 4
  7. 2024NeurIPS 2023 Competition: Privacy Preserving Federated Learning Document VQAcited by 2
  8. 2025dpmm: Differentially Private Marginal Models, a Library for Synthetic Tabular Data Generationcited by 2
  9. 2026Rethinking Anonymity Claims in Synthetic Data Generation: A Model-Centric Privacy Attack Perspectivecited by 1
  10. 2025Understanding the Impact of Data Domain Extraction on Synthetic Data Privacycited by 0
  11. 2026Tight Auditing of Differential Privacy in MST and AIMcited by 0
  12. 2026A Benchmark for Strategic Auditee Gaming Under Continuous Compliance Monitoringcited by 0