# Does Training with Synthetic Data Truly Protect Privacy?

arXiv: https://arxiv.org/abs/2502.12976 · HTML: https://mlsanity.com/paper/2502.12976/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 46 references that are in the map.
- First posted on arXiv 18 Feb 2025.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models](https://mlsanity.com/paper/2212.03860/) (2022)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021)
- [Unlocking High-Accuracy Differentially Private Image Classification through Scale](https://mlsanity.com/paper/2204.13650/) (2022)
- [Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory](https://mlsanity.com/paper/2211.10586/) (2022)
- [A Self-Supervised Descriptor for Image Copy Detection](https://mlsanity.com/paper/2202.10261/) (2022)
- [DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning](https://mlsanity.com/paper/2204.08499/) (2022)
- [Data-Free Knowledge Distillation for Deep Neural Networks](https://mlsanity.com/paper/1710.07535/) (2017)
- [Privacy for Free: How does Dataset Condensation Help Privacy?](https://mlsanity.com/paper/2206.00240/) (2022)
- [Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching](https://mlsanity.com/paper/2310.05773/) (2023)
- [TARGET: Federated Class-Continual Learning via Exemplar-Free Distillation](https://mlsanity.com/paper/2303.06937/) (2023)
- [Accelerating Dataset Distillation via Model Augmentation](https://mlsanity.com/paper/2212.06152/) (2022)
- [TAN Without a Burn: Scaling Laws of DP-SGD](https://mlsanity.com/paper/2210.03403/) (2022)
- [Introduction to Core-sets: an Updated Survey](https://mlsanity.com/paper/2011.09384/) (2020)
- [The 2021 Image Similarity Dataset and Challenge](https://mlsanity.com/paper/2106.09672/) (2021)
- [RelaxLoss: Defending Membership Inference Attacks without Losing Utility](https://mlsanity.com/paper/2207.05801/) (2022)
- [Real-Fake: Effective Training Data Synthesis Through Distribution Matching](https://mlsanity.com/paper/2310.10402/) (2023)

## Built on by (most cited first, 12 of 12)

- [AI Agent Behavioral Science](https://mlsanity.com/paper/2506.06366/) (2025)
- [FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2510.20250/) (2025)
- [LCMem: A Universal Model for Robust Image Memorization Detection](https://mlsanity.com/paper/2512.14421/) (2025)
- [CompLeak: Deep Learning Model Compression Exacerbates Privacy Leakage](https://mlsanity.com/paper/2507.16872/) (2025)
- [CoLA: A Choice Leakage Attack Framework to Expose Privacy Risks in Subset Training](https://mlsanity.com/paper/2604.12342/) (2026)
- [Decoupling Generalizability and Membership Privacy Risks in Neural Networks](https://mlsanity.com/paper/2602.02296/) (2026)
- [RedacBench: Can AI Erase Your Secrets?](https://mlsanity.com/paper/2603.20208/) (2026)
- [Pretraining Data Exposure in Large Language Models: A Survey of Membership Inference, Data Contamination, and Security Implications](https://mlsanity.com/paper/2605.26133/) (2026)
- [Studying the Role of Synthetic Data for Machine Learning-based Wireless Networks Traffic Forecasting](https://mlsanity.com/paper/2601.07646/) (2026)
- [Privacy Amplification Persists under Unlimited Synthetic Data Release](https://mlsanity.com/paper/2602.04895/) (2026)
- [Learnability and Privacy Vulnerability are Entangled in a Few Critical Weights](https://mlsanity.com/paper/2603.13186/) (2026)
- [Causal-Privacy Audit Workflow for Synthetic and Distilled Data in Dropout Support](https://mlsanity.com/paper/2606.15940/) (2026)

## Abstract (arXiv)

As synthetic data becomes increasingly popular in machine learning tasks, numerous methods--without formal differential privacy guarantees--use synthetic data for training. These methods often claim, either explicitly or implicitly, to protect the privacy of the original training data. In this work, we explore four different training paradigms: coreset selection, dataset distillation, data-free knowledge distillation, and synthetic data generated from diffusion models. While all these methods utilize synthetic data for training, they lead to vastly different conclusions regarding privacy preservation. We caution that empirical approaches to preserving data privacy require careful and rigorous evaluation; otherwise, they risk providing a false sense of privacy.