mlsanityData to 7 Oct 2026

Personalized Federated Training of Diffusion Models with Privacy Guarantees

arXiv 2504.00952 ↗First posted 1 Apr 2025cs.LG cs.AI cs.CVcited by 0

On the map: Diffusion models → Robustness, privacy and adversarial aspects

Abstract (arXiv)

The scarcity of accessible, compliant, and ethically sourced data presents a considerable challenge to the adoption of artificial intelligence (AI) in sensitive fields like healthcare, finance, and biomedical research. Furthermore, access to unrestricted public datasets is increasingly constrained due to rising concerns over privacy, copyright, and competition. Synthetic data has emerged as a promising alternative, and diffusion models -- a cutting-edge generative AI technology -- provide an effective solution for generating high-quality and diverse synthetic data. In this paper, we introduce a novel federated learning framework for training diffusion models on decentralized private datasets. Our framework leverages personalization and the inherent noise in the forward diffusion process to produce high-quality samples while ensuring robust differential privacy guarantees. Our experiments show that our framework outperforms non-collaborative training methods, particularly in settings with high data heterogeneity, and effectively reduces biases and imbalances in synthetic data, resulting in fairer downstream models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models. Builds on 70 of its 105 references that are in the map. First posted on arXiv 1 Apr 2025.

Builds on

70 of 105 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025RAPID: Retrieval Augmented Training of Differentially Private Diffusion Models18 Feb
  2. 2024FedBiP: Heterogeneous One-Shot Federated Learning with Personalized Latent Diffusion Models7 Oct
  3. 2024FedDM: Enhancing Communication Efficiency and Handling Data Heterogeneity in Federated Diffusion Models20 Jul
  4. 2024Training Diffusion Models with Federated Learning18 Jun
  5. 2024Differentially Private Fine-Tuning of Diffusion Models3 Jun
  6. 2024Navigating Heterogeneity and Privacy in One-Shot Federated Learning with Diffusion Models2 May
  7. 2024FedTabDiff: Federated Learning of Diffusion Probabilistic Models for Synthetic Mixed-Type Tabular Data Generation11 Jan
  8. 2023White-box Membership Inference Attacks against Diffusion Models11 Aug
  9. 2023Phoenix: A Federated Generative Diffusion Model7 Jun
  10. 2023An Efficient Membership Inference Attack for the Diffusion Model by Proximal Initialization26 May
  11. 2023DP-LDMs: Differentially Private Latent Diffusion Models25 May
  12. 2023Differentially Private Diffusion Models Generate Useful Synthetic Images27 Feb
3 more
  1. 2023Extracting Training Data from Diffusion Models30 Jan
  2. 2022How to Backdoor Diffusion Models?11 Dec
  3. 2022Differentially Private Diffusion Models18 Oct

Other references · 55

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  10. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  11. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  12. 2019Advances and Open Problems in Federated Learningcited by 3,021
38 more
  1. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  2. 2016Deep Learning with Differential Privacycited by 2,682
  3. 2019Invariant Risk Minimizationcited by 1,832
  4. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  5. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  6. 2016Pixel Recurrent Neural Networkscited by 814
  7. 2020The Future of Digital Health with Federated Learningcited by 625
  8. 2019Privacy-preserving Federated Brain Tumour Segmentationcited by 169
  9. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 169
  10. 2022Generative Modelling With Inverse Heat Dissipationcited by 164
  11. 2023DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesiscited by 159
  12. 2023On Memorization in Diffusion Modelscited by 122
  13. 2020GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generatorscited by 117
  14. 2020Revisiting Membership Inference Under Realistic Assumptionscited by 108
  15. 2025Masked Autoencoders Are Effective Tokenizers for Diffusion Modelscited by 93
  16. 2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 86
  17. 2019Generative Models for Effective ML on Private, Decentralized Datasetscited by 85
  18. 2021Federated Evaluation and Tuning for On-Device Personalization: System Design & Applicationscited by 84
  19. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  20. 2024On the Societal Impact of Open Foundation Modelscited by 64
  21. 2023Differentially Private Synthetic Data via Foundation Model APIs 1: Imagescited by 63
  22. 2020Federated Learning for Breast Density Classification: A Real-World Implementationcited by 62
  23. 2024Learning general Gaussian mixtures with efficient score matchingcited by 43
  24. 2024Learning Mixtures of Gaussians Using Diffusion Modelscited by 41
  25. 2022Progressive Deblurring of Diffusion Models for Coarse-to-Fine Image Synthesiscited by 34
  26. 2024DiffuseHigh: Training-free Progressive High-Resolution Image Synthesis through Structure Guidancecited by 33
  27. 2022Reconstructing Training Data from Model Gradient, Provablycited by 27
  28. 2022Private Non-Convex Federated Learning Without a Trusted Servercited by 26
  29. 2023Coarse-to-Fine: a Hierarchical Diffusion Model for Molecule Generation in 3Dcited by 24
  30. 2024Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Modelscited by 24
  31. 2022Personalization Improves Privacy-Accuracy Tradeoffs in Federated Learningcited by 23
  32. 2024On the Feature Learning in Diffusion Modelscited by 19
  33. 2024Step-by-Step Diffusion: An Elementary Tutorialcited by 16
  34. 2025Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?cited by 15
  35. 2024Why Does Differential Privacy with Large Epsilon Defend Against Practical Membership Inference Attacks?cited by 12
  36. 2023DIFF2: Differential Private Optimization via Gradient Differences for Nonconvex Distributed Learningcited by 10
  37. 2024Slight Corruption in Pre-training Data Makes Better Diffusion Modelscited by 9
  38. 2023Partially Personalized Federated Learning: Breaking the Curse of Data Heterogeneitycited by 8

50 most cited of 55 All 55 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Leveraging Programmatically Generated Synthetic Data for Differentially Private Diffusion Training
  2. 2023Federated Learning with Diffusion Models for Privacy-Sensitive Vision Tasks
  3. 2024PDFed: Privacy-Preserving and Decentralized Asynchronous Federated Learning for Diffusion Models
  4. 2024Federated Diffusion Modeling with Differential Privacy for Tabular Data Synthesis
  5. 2025Does Training with Synthetic Data Truly Protect Privacy?
  6. 2025Enabling PSO-Secure Synthetic Data Sharing Using Diversity-Aware Diffusion Models
  7. 2024Private Synthetic Text Generation with Diffusion Models
  8. 2022Federated Learning with GAN-based Data Synthesis for Non-IID Clients