# Training Diffusion Models with Federated Learning

arXiv: https://arxiv.org/abs/2406.12575 · HTML: https://mlsanity.com/paper/2406.12575/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 33 of its 43 references that are in the map.
- First posted on arXiv 18 Jun 2024.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [Federated Learning for Mobile Keyboard Prediction](https://mlsanity.com/paper/1811.03604/) (2018)
- [Federated Learning on Non-IID Data Silos: An Experimental Study](https://mlsanity.com/paper/2102.02079/) (2021)
- [A Kernelized Stein Discrepancy for Goodness-of-fit Tests and Model Evaluation](https://mlsanity.com/paper/1602.03253/) (2016)
- [Distributed Mean Estimation with Limited Communication](https://mlsanity.com/paper/1611.00429/) (2016)
- [FedGAN: Federated Generative Adversarial Networks for Distributed Data](https://mlsanity.com/paper/2006.07228/) (2020)
- [Copyright in Generative Deep Learning](https://mlsanity.com/paper/2105.09266/) (2021)
- [Federated Generative Adversarial Learning](https://mlsanity.com/paper/2005.03793/) (2020)

## Built on by (most cited first, 15 of 15)

- [Deep Generative Models for 3D Medical Image Synthesis](https://mlsanity.com/paper/2410.17664/) (2024)
- [Federated Learning for Diffusion Models](https://mlsanity.com/paper/2503.06426/) (2025)
- [FedDM: Enhancing Communication Efficiency and Handling Data Heterogeneity in Federated Diffusion Models](https://mlsanity.com/paper/2407.14730/) (2024)
- [CollaFuse: Collaborative Diffusion Models](https://mlsanity.com/paper/2406.14429/) (2024)
- [Federated Diffusion Modeling with Differential Privacy for Tabular Data Synthesis](https://mlsanity.com/paper/2412.16083/) (2024)
- [FedCGD: Collective Gradient Divergence Optimized Scheduling for Wireless Federated Learning](https://mlsanity.com/paper/2506.07581/) (2025)
- [FedSDA: Federated Stain Distribution Alignment for Non-IID Histopathological Image Classification](https://mlsanity.com/paper/2511.12044/) (2025)
- [Dynamic Topology Optimization for Non-IID Data in Decentralized Learning](https://mlsanity.com/paper/2602.03383/) (2026)
- [Fed-NDIF: A Noise-Embedded Federated Diffusion Model For Low-Count Whole-Body PET Denoising](https://mlsanity.com/paper/2503.16635/) (2025)
- [FedPhD: Federated Pruning with Hierarchical Learning of Diffusion Models](https://mlsanity.com/paper/2507.06449/) (2025)
- [On the Tradeoffs of On-Device Generative Models in Federated Predictive Maintenance Systems](https://mlsanity.com/paper/2605.07860/) (2026)
- [Personalized Federated Training of Diffusion Models with Privacy Guarantees](https://mlsanity.com/paper/2504.00952/) (2025)
- [Federated Flow Matching](https://mlsanity.com/paper/2509.21250/) (2025)
- [UMEDA: Unified Multi-modal Efficient Data Fusion for Privacy-Preserving Graph Federated Learning via Spectral-Gated Attention and Diffusion-Based Operator Alignment](https://mlsanity.com/paper/2605.08288/) (2026)
- [Compositional Generative Modeling from Decentralized Data](https://mlsanity.com/paper/2606.10153/) (2026)

## Abstract (arXiv)

The training of diffusion-based models for image generation is predominantly controlled by a select few Big Tech companies, raising concerns about privacy, copyright, and data authority due to their lack of transparency regarding training data. To ad-dress this issue, we propose a federated diffusion model scheme that enables the independent and collaborative training of diffusion models without exposing local data. Our approach adapts the Federated Averaging (FedAvg) algorithm to train a Denoising Diffusion Model (DDPM). Through a novel utilization of the underlying UNet backbone, we achieve a significant reduction of up to 74% in the number of parameters exchanged during training,compared to the naive FedAvg approach, whilst simultaneously maintaining image quality comparable to the centralized setting, as evaluated by the FID score.