mlsanityData to 14 Oct 2026

DPImageBench: A Unified Benchmark for Differentially Private Image Synthesis

arXiv 2503.14681 ↗First posted 18 Mar 2025cs.CR cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Differentially private (DP) image synthesis aims to generate artificial images that retain the properties of sensitive images while protecting the privacy of individual images within the dataset. Despite recent advancements, we find that inconsistent--and sometimes flawed--evaluation protocols have been applied across studies. This not only impedes the understanding of current methods but also hinders future advancements. To address the issue, this paper introduces DPImageBench for DP image synthesis, with thoughtful design across several dimensions: (1) Methods. We study eleven prominent methods and systematically characterize each based on model architecture, pretraining strategy, and privacy mechanism. (2) Evaluation. We include nine datasets and seven fidelity and utility metrics to thoroughly assess them. Notably, we find that a common practice of selecting downstream classifiers based on the highest accuracy on the sensitive test set not only violates DP but also overestimates the utility scores. DPImageBench corrects for these mistakes. (3) Platform. Despite the methods and evaluation protocols, DPImageBench provides a standardized interface that accommodates current and future implementations within a unified framework. With DPImageBench, we have several noteworthy findings. For example, contrary to the common wisdom that pretraining on public image datasets is usually beneficial, we find that the distributional similarity between pretraining and sensitive images significantly impacts the performance of the synthetic images and does not always yield improvements. In addition, adding noise to low-dimensional features, such as the high-level characteristics of sensitive images, is less affected by the privacy budget compared to adding noise to high-dimensional features, like weight gradients. The former methods perform better than the latter under a low privacy budget.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 62 of its 94 references that are in the map. First posted on arXiv 18 Mar 2025.

Builds on

62 of 94 references.

References · 62

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  8. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  11. 2016Wide Residual Networkscited by 3,070
  12. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
38 more
  1. 2016Improved Techniques for Training GANscited by 2,970
  2. 2016Deep Learning with Differential Privacycited by 2,728
  3. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  4. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  5. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  6. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  7. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  8. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  9. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  10. 2017Renyi Differential Privacycited by 650
  11. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  12. 2016NIPS 2016 Tutorial: Generative Adversarial Networkscited by 285
  13. 2018Differentially Private Generative Adversarial Networkcited by 235
  14. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
  15. 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 218
  16. 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 204
  17. 2017VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learningcited by 200
  18. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 199
  19. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 153
  20. 2023Privacy Auditing with One (1) Training Runcited by 123
  21. 2022Differentially Private Diffusion Modelscited by 119
  22. 2020GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generatorscited by 117
  23. 2022Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipecited by 108
  24. 2017PacGAN: The power of two samples in generative adversarial networkscited by 104
  25. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 96
  26. 2022AIM: An Adaptive and Iterative Mechanism for Differentially Private Synthetic Datacited by 90
  27. 2023Membership Inference Attacks against Diffusion Modelscited by 86
  28. 2023Differentially Private Diffusion Models Generate Useful Synthetic Imagescited by 86
  29. 2021Benchmarking Differentially Private Synthetic Data Generation Algorithmscited by 81
  30. 2020DP-MERF: Differentially Private Mean Embeddings with Random Features for Practical Privacy-Preserving Data Generationcited by 78
  31. 2024Differentially Private Synthetic Data via Foundation Model APIs 2: Textcited by 76
  32. 2023Differentially Private Synthetic Data via Foundation Model APIs 1: Imagescited by 65
  33. 2023SoK: Privacy-Preserving Data Synthesiscited by 38
  34. 2023DP-LDMs: Differentially Private Latent Diffusion Modelscited by 29
  35. 2022Pre-trained Perceptual Features Improve Differentially Private Image Generationcited by 28
  36. 2023Feature Likelihood Divergence: Evaluating the Generalization of Generative Models Using Samplescited by 25
  37. 2023Selective Pre-training for Private Fine-tuningcited by 23
  38. 2020Why Spectral Normalization Stabilizes GANs: Analysis and Improvementscited by 21

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 5.

11 papers.

  1. 2025Synthesize Privacy-Preserving High-Resolution Images via Private Textual Intermediariescited by 8
  2. 2025From Easy to Hard: Building a Shortcut for Differentially Private Image Synthesiscited by 7
  3. 2025Struct-Bench: A Benchmark for Differentially Private Structured Text Generationcited by 7
  4. 2025Private Evolution Convergescited by 2
  5. 2025PrivORL: Differentially Private Synthetic Dataset for Offline Reinforcement Learningcited by 1
  6. 2025Beyond One-Size-Fits-All: Neural Networks for Differentially Private Tabular Data Synthesiscited by 0
  7. 2026From Easy to Hard++: Promoting Differentially Private Image Synthesis Through Spatial-Frequency Curriculumcited by 0
  8. 2026ContinuousBench: Can Differentially Private Synthetic Text Improve Capabilities?cited by 0
  9. 2026Collaborative Synthetic Data Generation for Knowledge Transfer in Federated Learningcited by 0
  10. 2026Differentially Private Paired Table-Image Multimodal Synthesiscited by 0
  11. 2026QuanText: Protecting Dataset-Level Secrets in Textual Data Sharingcited by 0