mlsanityData to 7 Oct 2026

Efficient Differentially Private Fine-Tuning of Diffusion Models

arXiv 2406.05257 ↗First posted 7 Jun 2024cs.LG cs.CRcited by 1

On the map: Diffusion models → Robustness, privacy and adversarial aspects

Abstract (arXiv)

The recent developments of Diffusion Models (DMs) enable generation of astonishingly high-quality synthetic samples. Recent work showed that the synthetic samples generated by the diffusion model, which is pre-trained on public data and fully fine-tuned with differential privacy on private data, can train a downstream classifier, while achieving a good privacy-utility tradeoff. However, fully fine-tuning such large diffusion models with DP-SGD can be very resource-demanding in terms of memory usage and computation. In this work, we investigate Parameter-Efficient Fine-Tuning (PEFT) of diffusion models using Low-Dimensional Adaptation (LoDA) with Differential Privacy. We evaluate the proposed method with the MNIST and CIFAR-10 datasets and demonstrate that such efficient fine-tuning can also generate useful synthetic samples for training downstream classifiers, with guaranteed privacy protection of fine-tuning data. Our source code will be made available on GitHub.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models. Builds on 16 of its 24 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

16 of 24 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Differentially Private Diffusion Models Generate Useful Synthetic Images27 Feb
  2. 2023Extracting Training Data from Diffusion Models30 Jan

Other references · 14

  1. 2022Classifier-Free Diffusion Guidancecited by 5,717
  2. 2016Deep Learning with Differential Privacycited by 2,682
  3. 2019An Introduction to Variational Autoencoderscited by 997
  4. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  5. 2018Differentially Private Generative Adversarial Networkcited by 235
  6. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  7. 2021Unlearnable Examples: Making Personal Data Unexploitablecited by 188
  8. 2020GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generatorscited by 117
  9. 2020DP-MERF: Differentially Private Mean Embeddings with Random Features for Practical Privacy-Preserving Data Generationcited by 78
  10. 2022Differentially Private Optimization on Large Model at Small Costcited by 65
  11. 2021Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergencecited by 54
  12. 2022Pre-trained Perceptual Features Improve Differentially Private Image Generationcited by 28
2 more
  1. 2021PEARL: Data Synthesis via Private Embeddings and Adversarial Reconstruction Learningcited by 23
  2. 2022DPD-fVAE: Synthetic Data Generation Using Federated Variational Autoencoders With Differentially-Private Decodercited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026A Data-Interventional Framework for Auditing Privacy and Fairness in Generative Medical Imagingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Differentially Private Fine-Tuning of Diffusion Models
  2. 2022Differentially Private Diffusion Models
  3. 2023On the Inherent Privacy Properties of Discrete Denoising Diffusion Models
  4. 2024Leveraging Programmatically Generated Synthetic Data for Differentially Private Diffusion Training
  5. 2023DP-LDMs: Differentially Private Latent Diffusion Models
  6. 2024Shake to Leak: Fine-tuning Diffusion Models Can Amplify the Generative Privacy Risk
  7. 2024Learning Differentially Private Diffusion Models via Stochastic Adversarial Distillation
  8. 2024Risks When Sharing LoRA Fine-Tuned Diffusion Model Weights