mlsanityData to 7 Oct 2026

Does Differentially Private Synthetic Data Lead to Synthetic Discoveries?

arXiv 2403.13612 ↗First posted 20 Mar 2024cs.LG stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

Background: Synthetic data has been proposed as a solution for sharing anonymized versions of sensitive biomedical datasets. Ideally, synthetic data should preserve the structure and statistical properties of the original data, while protecting the privacy of the individual subjects. Differential privacy (DP) is currently considered the gold standard approach for balancing this trade-off. Objectives: To investigate the reliability of group differences identified by independent sample tests on DP-synthetic data. The evaluation is conducted in terms of the tests' Type I and Type II errors. The former quantifies the tests' validity i.e. whether the probability of false discoveries is indeed below the significance level, and the latter indicates the tests' power in making real discoveries. Methods: We evaluate the Mann-Whitney U test, Student's t-test, chi-squared test and median test on DP-synthetic data. The private synthetic datasets are generated from real-world data, including a prostate cancer dataset (n=500) and a cardiovascular dataset (n=70 000), as well as on bivariate and multivariate simulated data. Five different DP-synthetic data generation methods are evaluated, including two basic DP histogram release methods and MWEM, Private-PGM, and DP GAN algorithms. Conclusion: A large portion of the evaluation results expressed dramatically inflated Type I errors, especially at privacy budget levels of ε≤ 1. This result calls for caution when releasing and analyzing DP-synthetic data: low p-values may be obtained in statistical tests simply as a byproduct of the noise added to protect privacy. A DP smoothed histogram-based synthetic data generation method was shown to produce valid Type I error for all privacy levels tested but required a large original dataset size and a modest privacy budget (ε≥ 5) in order to have reasonable Type II error.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 21 of its 76 references that are in the map. First posted on arXiv 20 Mar 2024.

Builds on

21 of 76 references.

References · 21

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2022Generative Adversarial Networkscited by 5,603
  3. 2016Deep Learning with Differential Privacycited by 2,682
  4. 2017Improved Training of Wasserstein GANscited by 2,223
  5. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  6. 2016NIPS 2016 Tutorial: Generative Adversarial Networkscited by 284
  7. 2020The Discrete Gaussian for Differential Privacycited by 134
  8. 2020GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generatorscited by 117
  9. 2020Synthetic Data -- Anonymisation Groundhog Daycited by 105
  10. 2021Winning the NIST Contest: A scalable and general approach to differentially private synthetic datacited by 101
  11. 2022AIM: An Adaptive and Iterative Mechanism for Differentially Private Synthetic Datacited by 88
  12. 2022Synthetic Data -- what, why and how?cited by 66
9 more
  1. 2016Differentially Private Chi-Squared Hypothesis Testing: Goodness of Fit and Independence Testingcited by 24
  2. 2020Parametric Bootstrap for Differentially Private Confidence Intervalscited by 19
  3. 2016Comparative Study of Differentially Private Data Synthesis Methodscited by 13
  4. 2019Differentially Private Nonparametric Hypothesis Testingcited by 12
  5. 2020Really Useful Synthetic Data -- A Framework to Evaluate the Quality of Differentially Private Synthetic Datacited by 11
  6. 2022Noise-Aware Statistical Inference with Differentially Private Synthetic Datacited by 9
  7. 2022Faking feature importance: A cautionary tale on the use of differentially-private synthetic datacited by 8
  8. 2021Covariance's Loss is Privacy's Gain: Computationally Efficient, Private and Accurate Synthetic Datacited by 7
  9. 2018pMSE Mechanism: Differentially Private Synthetic Data with Maximal Distributional Similaritycited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025A Survey on Deep Learning Approaches for Tabular Data Generation: Utility, Alignment, Fidelity, Privacy, Diversity, and Beyondcited by 13
  2. 2026Noise-Calibrated Inference from Differentially Private Sufficient Statistics in Exponential Familiescited by 0
  3. 2026Synthetic Data in Education: Empirical Insights from Traditional Resampling and Deep Generative Modelscited by 0
  4. 2026Workload-Preserving Differentially Private Synthetic Data for Causal Inference via Maximum-Entropy Calibrationcited by 0