# Does Differentially Private Synthetic Data Lead to Synthetic Discoveries?

arXiv: https://arxiv.org/abs/2403.13612 · HTML: https://mlsanity.com/paper/2403.13612/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 76 references that are in the map.
- First posted on arXiv 20 Mar 2024.

## Builds on (most cited first, 21 of 21)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [NIPS 2016 Tutorial: Generative Adversarial Networks](https://mlsanity.com/paper/1701.00160/) (2016)
- [The Discrete Gaussian for Differential Privacy](https://mlsanity.com/paper/2004.00010/) (2020)
- [GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generators](https://mlsanity.com/paper/2006.08265/) (2020)
- [Synthetic Data -- Anonymisation Groundhog Day](https://mlsanity.com/paper/2011.07018/) (2020)
- [Winning the NIST Contest: A scalable and general approach to differentially private synthetic data](https://mlsanity.com/paper/2108.04978/) (2021)
- [AIM: An Adaptive and Iterative Mechanism for Differentially Private Synthetic Data](https://mlsanity.com/paper/2201.12677/) (2022)
- [Synthetic Data -- what, why and how?](https://mlsanity.com/paper/2205.03257/) (2022)
- [Differentially Private Chi-Squared Hypothesis Testing: Goodness of Fit and Independence Testing](https://mlsanity.com/paper/1602.03090/) (2016)
- [Parametric Bootstrap for Differentially Private Confidence Intervals](https://mlsanity.com/paper/2006.07749/) (2020)
- [Comparative Study of Differentially Private Data Synthesis Methods](https://mlsanity.com/paper/1602.01063/) (2016)
- [Differentially Private Nonparametric Hypothesis Testing](https://mlsanity.com/paper/1903.09364/) (2019)
- [Really Useful Synthetic Data -- A Framework to Evaluate the Quality of Differentially Private Synthetic Data](https://mlsanity.com/paper/2004.07740/) (2020)
- [Noise-Aware Statistical Inference with Differentially Private Synthetic Data](https://mlsanity.com/paper/2205.14485/) (2022)
- [Faking feature importance: A cautionary tale on the use of differentially-private synthetic data](https://mlsanity.com/paper/2203.01363/) (2022)
- [Covariance's Loss is Privacy's Gain: Computationally Efficient, Private and Accurate Synthetic Data](https://mlsanity.com/paper/2107.05824/) (2021)
- [pMSE Mechanism: Differentially Private Synthetic Data with Maximal Distributional Similarity](https://mlsanity.com/paper/1805.09392/) (2018)

## Built on by (most cited first, 4 of 4)

- [A Survey on Deep Learning Approaches for Tabular Data Generation: Utility, Alignment, Fidelity, Privacy, Diversity, and Beyond](https://mlsanity.com/paper/2503.05954/) (2025)
- [Noise-Calibrated Inference from Differentially Private Sufficient Statistics in Exponential Families](https://mlsanity.com/paper/2603.02010/) (2026)
- [Synthetic Data in Education: Empirical Insights from Traditional Resampling and Deep Generative Models](https://mlsanity.com/paper/2604.21031/) (2026)
- [Workload-Preserving Differentially Private Synthetic Data for Causal Inference via Maximum-Entropy Calibration](https://mlsanity.com/paper/2607.08122/) (2026)

## Abstract (arXiv)

Background: Synthetic data has been proposed as a solution for sharing anonymized versions of sensitive biomedical datasets. Ideally, synthetic data should preserve the structure and statistical properties of the original data, while protecting the privacy of the individual subjects. Differential privacy (DP) is currently considered the gold standard approach for balancing this trade-off. Objectives: To investigate the reliability of group differences identified by independent sample tests on DP-synthetic data. The evaluation is conducted in terms of the tests' Type I and Type II errors. The former quantifies the tests' validity i.e. whether the probability of false discoveries is indeed below the significance level, and the latter indicates the tests' power in making real discoveries. Methods: We evaluate the Mann-Whitney U test, Student's t-test, chi-squared test and median test on DP-synthetic data. The private synthetic datasets are generated from real-world data, including a prostate cancer dataset (n=500) and a cardiovascular dataset (n=70 000), as well as on bivariate and multivariate simulated data. Five different DP-synthetic data generation methods are evaluated, including two basic DP histogram release methods and MWEM, Private-PGM, and DP GAN algorithms. Conclusion: A large portion of the evaluation results expressed dramatically inflated Type I errors, especially at privacy budget levels of ε≤ 1. This result calls for caution when releasing and analyzing DP-synthetic data: low p-values may be obtained in statistical tests simply as a byproduct of the noise added to protect privacy. A DP smoothed histogram-based synthetic data generation method was shown to produce valid Type I error for all privacy levels tested but required a large original dataset size and a modest privacy budget (ε≥ 5) in order to have reasonable Type II error.