mlsanityData to 14 Oct 2026

DP-LDMs: Differentially Private Latent Diffusion Models

arXiv 2305.15759 ↗First posted 25 May 2023stat.ML cs.CR cs.LGcited by 29

On the map: Diffusion models → Robustness, privacy and adversarial aspects

Abstract (arXiv)

Diffusion models (DMs) are one of the most widely used generative models for producing high quality images. However, a flurry of recent papers points out that DMs are least private forms of image generators, by extracting a significant number of near-identical replicas of training images from DMs. Existing privacy-enhancing techniques for DMs, unfortunately, do not provide a good privacy-utility tradeoff. In this paper, we aim to improve the current state of DMs with differential privacy (DP) by adopting the textitLatent Diffusion Models (LDMs). LDMs are equipped with powerful pre-trained autoencoders that map the high-dimensional pixels into lower-dimensional latent representations, in which DMs are trained, yielding a more efficient and fast training of DMs. Rather than fine-tuning the entire LDMs, we fine-tune only the textitattention modules of LDMs with DP-SGD, reducing the number of trainable parameters by roughly 90% and achieving a better privacy-accuracy trade-off. Our approach allows us to generate realistic, high-dimensional images (256x256) conditioned on text prompts with DP guarantees, which, to the best of our knowledge, has not been attempted before. Our approach provides a promising direction for training more powerful, yet training-efficient differentially private DMs, producing high-quality DP images. Our code is available at https://anonymous.4open.science/r/DP-LDM-4525.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models. Builds on 53 of its 81 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

53 of 81 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Differentially Private Diffusion Models Generate Useful Synthetic Images27 Feb
  2. 2023Membership Inference Attacks against Diffusion Models7 Feb
  3. 2023Are Diffusion Models Vulnerable to Membership Inference Attacks?2 Feb
  4. 2023Extracting Training Data from Diffusion Models30 Jan
  5. 2023Membership Inference of Diffusion Models24 Jan
  6. 2022Differentially Private Diffusion Models18 Oct

Other references · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  10. 2022Generative Adversarial Networkscited by 5,627
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  12. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
35 more
  1. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  2. 2016Wide Residual Networkscited by 3,070
  3. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  4. 2016Deep Learning with Differential Privacycited by 2,728
  5. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  6. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  7. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  8. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 424
  9. 2018Deep Learning for Classical Japanese Literaturecited by 422
  10. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  11. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  12. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 383
  13. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 348
  14. 2018Subsampled R'enyi Differential Privacy and Analytical Moments Accountantcited by 236
  15. 2018Differentially Private Generative Adversarial Networkcited by 235
  16. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
  17. 2020DP-CGAN: Differentially Private Synthetic Data and Label Generationcited by 118
  18. 2020GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generatorscited by 117
  19. 2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 88
  20. 2020DP-MERF: Differentially Private Mean Embeddings with Random Features for Practical Privacy-Preserving Data Generationcited by 78
  21. 2022Membership Inference Attacks Against Text-to-image Generation Modelscited by 69
  22. 2023Differentially Private Synthetic Data via Foundation Model APIs 1: Imagescited by 65
  23. 2021Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergencecited by 54
  24. 2022Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacycited by 48
  25. 2022Shape-Guided Diffusion with Inside-Outside Attentioncited by 45
  26. 2017Differentially Private Mixture of Generative Neural Networkscited by 39
  27. 2019Differentially Private Generative Adversarial Networks for Time Series, Continuous, and Discrete Open Datacited by 31
  28. 2022Pre-trained Perceptual Features Improve Differentially Private Image Generationcited by 28
  29. 2021PEARL: Data Synthesis via Private Embeddings and Adversarial Reconstruction Learningcited by 23
  30. 2021Hermite Polynomial Features for Private Data Generationcited by 21
  31. 2023Private GANs, Revisitedcited by 17
  32. 2023Differentially Private Neural Tangent Kernels for Privacy-Preserving Data Generationcited by 15
  33. 2022DPD-fVAE: Synthetic Data Generation Using Federated Variational Autoencoders With Differentially-Private Decodercited by 14
  34. 2022DP2-VAE: Differentially Private Pre-trained Variational Autoencoderscited by 13
  35. 2018pMSE Mechanism: Differentially Private Synthetic Data with Maximal Distributional Similaritycited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 9; 2025: 10; 2026: 4.

29 papers.

Sort:

By area:

  1. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 60
  2. 2023AutoDiff: combining Auto-encoder and Diffusion model for tabular data synthesizingcited by 24
  3. 2025Training-Free Safe Denoisers for Safe Use of Diffusion Modelscited by 21
  4. 2024Differentially Private Fine-Tuning of Diffusion Modelscited by 20
  5. 2023PrivImage: Differentially Private Synthetic Image Generation using Diffusion Models with Semantic-Aware Pretrainingcited by 13
  6. 2025Differentially Private Synthetic Data via APIs 3: Using Simulators Instead of Foundation Modelcited by 13
  7. 2024Replication in Visual Diffusion Models: A Survey and Outlookcited by 11
  8. 2025DPImageBench: A Unified Benchmark for Differentially Private Image Synthesiscited by 11
  9. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  10. 2025Synthesize Privacy-Preserving High-Resolution Images via Private Textual Intermediariescited by 8
  11. 2025From Easy to Hard: Building a Shortcut for Differentially Private Image Synthesiscited by 7
  12. 2024TimeAutoDiff: A Unified Framework for Generation, Imputation, Forecasting, and Time-Varying Metadata Conditioning of Heterogeneous Time Series Tabular Datacited by 5
17 more
  1. 2023A Unified View of Differentially Private Deep Generative Modelingcited by 4
  2. 2024DP-RDM: Adapting Diffusion Models to Private Domains Without Fine-Tuningcited by 4
  3. 2025RAPID: Retrieval Augmented Training of Differentially Private Diffusion Modelscited by 3
  4. 2026Scalable Differentially Private Data Compression via Diffusion and Stochastic Codescited by 3
  5. 2024Learning Differentially Private Diffusion Models via Stochastic Adversarial Distillationcited by 2
  6. 2023Quantum Solutions to the Privacy vs. Utility Tradeoffcited by 1
  7. 2023PAC Privacy Preserving Diffusion Modelscited by 1
  8. 2024Bayesian Power Steering: An Effective Approach for Domain Adaptation of Diffusion Modelscited by 1
  9. 2024LoyalDiffusion: A Diffusion Model Guarding Against Data Replicationcited by 1
  10. 2025SoK: Can Synthetic Images Replace Real Data? A Survey of Utility and Privacy of Synthetic Image Generationcited by 1
  11. 2025Implementing Adaptations for Vision AutoRegressive Modelcited by 1
  12. 2025DP-DocLDM: Differentially Private Document Image Generation using Latent Diffusion Modelscited by 1
  13. 2024Expert Routing with Synthetic Data for Continual Learningcited by 0
  14. 2025Personalized Federated Training of Diffusion Models with Privacy Guaranteescited by 0
  15. 2026Privacy-Preserving Model Transcription with Differentially Private Synthetic Distillationcited by 0
  16. 2026Decomposing Private Image Generation via Coarse-to-Fine Wavelet Modelingcited by 0
  17. 2026Protecting Sensitive Data in Image Synthesis via PAC-Private Adaptation for Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DP-TLDM: Differentially Private Tabular Latent Diffusion Model
  2. 2024PID: Prompt-Independent Data Protection Against Latent Diffusion Models
  3. 2025Latent Diffusion Unlearning: Protecting Against Unauthorized Personalization Through Trajectory Shifted Perturbations
  4. 2023On the Inherent Privacy Properties of Discrete Denoising Diffusion Models
  5. 2024Gradient-Guided Conditional Diffusion Models for Private Image Reconstruction: Analyzing Adversarial Impacts of Differential Privacy and Denoising
  6. 2024Efficient Differentially Private Fine-Tuning of Diffusion Models
  7. 2024Differentially Private Adaptation of Diffusion Models via Noisy Aggregated Embeddings
  8. 2024Model Will Tell: Training Membership Inference for Diffusion Models