# From Easy to Hard: Building a Shortcut for Differentially Private Image Synthesis

arXiv: https://arxiv.org/abs/2504.01395 · HTML: https://mlsanity.com/paper/2504.01395/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 40 of its 62 references that are in the map.
- First posted on arXiv 2 Apr 2025.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)

## Builds on (most cited first, 30 of 40)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [R'enyi Differential Privacy of the Sampled Gaussian Mechanism](https://mlsanity.com/paper/1908.10530/) (2019)
- [Combined Scaling for Zero-shot Transfer Learning](https://mlsanity.com/paper/2111.10050/) (2021)
- [Numerical Composition of Differential Privacy](https://mlsanity.com/paper/2106.02848/) (2021)
- [Differentially Private Diffusion Models](https://mlsanity.com/paper/2210.09929/) (2022)
- [DP-CGAN: Differentially Private Synthetic Data and Label Generation](https://mlsanity.com/paper/2001.09700/) (2020)
- [GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generators](https://mlsanity.com/paper/2006.08265/) (2020)
- [Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe](https://mlsanity.com/paper/2210.14348/) (2022)
- [A Comprehensive Survey of Image Augmentation Techniques for Deep Learning](https://mlsanity.com/paper/2205.01491/) (2022)
- [AIM: An Adaptive and Iterative Mechanism for Differentially Private Synthetic Data](https://mlsanity.com/paper/2201.12677/) (2022)
- [Differentially Private Diffusion Models Generate Useful Synthetic Images](https://mlsanity.com/paper/2302.13861/) (2023)
- [Generative Models for Effective ML on Private, Decentralized Datasets](https://mlsanity.com/paper/1911.06679/) (2019)
- [DP-MERF: Differentially Private Mean Embeddings with Random Features for Practical Privacy-Preserving Data Generation](https://mlsanity.com/paper/2002.11603/) (2020)
- [Differentially Private Synthetic Data via Foundation Model APIs 2: Text](https://mlsanity.com/paper/2403.01749/) (2024)
- [Differentially Private Language Models for Secure Data Sharing](https://mlsanity.com/paper/2210.13918/) (2022)
- [Differentially Private Synthetic Data via Foundation Model APIs 1: Images](https://mlsanity.com/paper/2305.15560/) (2023)
- [G-PATE: Scalable Differentially Private Data Generator via Private Aggregation of Teacher Discriminators](https://mlsanity.com/paper/1906.09338/) (2019)
- [PPGAN: Privacy-preserving Generative Adversarial Network](https://mlsanity.com/paper/1910.02007/) (2019)
- [DataLens: Scalable Privacy Preserving Training via Gradient Compression and Aggregation](https://mlsanity.com/paper/2103.11109/) (2021)
- [DP-LDMs: Differentially Private Latent Diffusion Models](https://mlsanity.com/paper/2305.15759/) (2023)
- [Pre-trained Perceptual Features Improve Differentially Private Image Generation](https://mlsanity.com/paper/2205.12900/) (2022)
- [Differentially Private k-Means Clustering](https://mlsanity.com/paper/1504.05998/) (2015)

## Built on by (most cited first, 7 of 7)

- [DATABench: Evaluating Dataset Auditing in Deep Learning from an Adversarial Perspective](https://mlsanity.com/paper/2507.05622/) (2025)
- [PrivORL: Differentially Private Synthetic Dataset for Offline Reinforcement Learning](https://mlsanity.com/paper/2512.07342/) (2025)
- [Privacy-Utility Trade-off in Data Publication: A Bilevel Optimization Framework with Curvature-Guided Perturbation](https://mlsanity.com/paper/2509.02048/) (2025)
- [Beyond One-Size-Fits-All: Neural Networks for Differentially Private Tabular Data Synthesis](https://mlsanity.com/paper/2511.13893/) (2025)
- [From Easy to Hard++: Promoting Differentially Private Image Synthesis Through Spatial-Frequency Curriculum](https://mlsanity.com/paper/2601.06368/) (2026)
- [Differentially Private Paired Table-Image Multimodal Synthesis](https://mlsanity.com/paper/2609.00708/) (2026)
- [Protecting Sensitive Data in Image Synthesis via PAC-Private Adaptation for Diffusion Models](https://mlsanity.com/paper/2610.04038/) (2026)

## Abstract (arXiv)

Differentially private (DP) image synthesis aims to generate synthetic images from a sensitive dataset, alleviating the privacy leakage concerns of organizations sharing and utilizing synthetic images. Although previous methods have significantly progressed, especially in training diffusion models on sensitive images with DP Stochastic Gradient Descent (DP-SGD), they still suffer from unsatisfactory performance. In this work, inspired by curriculum learning, we propose a two-stage DP image synthesis framework, where diffusion models learn to generate DP synthetic images from easy to hard. Unlike existing methods that directly use DP-SGD to train diffusion models, we propose an easy stage in the beginning, where diffusion models learn simple features of the sensitive images. To facilitate this easy stage, we propose to use `central images', simply aggregations of random samples of the sensitive dataset. Intuitively, although those central images do not show details, they demonstrate useful characteristics of all images and only incur minimal privacy costs, thus helping early-phase model training. We conduct experiments to present that on the average of four investigated image datasets, the fidelity and utility metrics of our synthetic images are 33.1% and 2.1% better than the state-of-the-art method.