# Federated Learning with Diffusion Models for Privacy-Sensitive Vision Tasks

arXiv: https://arxiv.org/abs/2311.16538 · HTML: https://mlsanity.com/paper/2311.16538/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 13 of its 21 references that are in the map.
- First posted on arXiv 28 Nov 2023.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [LightGrad: Lightweight Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2308.16569/) (2023)
- [Toward Multiple Federated Learning Services Resource Sharing in Mobile Edge Networks](https://mlsanity.com/paper/2011.12469/) (2020)

## Built on by (most cited first, 10 of 10)

- [Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learning](https://mlsanity.com/paper/2409.01128/) (2024)
- [Deep Generative Models for 3D Medical Image Synthesis](https://mlsanity.com/paper/2410.17664/) (2024)
- [Federated Learning for Diffusion Models](https://mlsanity.com/paper/2503.06426/) (2025)
- [A Systematic Review of Federated Generative Models](https://mlsanity.com/paper/2405.16682/) (2024)
- [FedSDA: Federated Stain Distribution Alignment for Non-IID Histopathological Image Classification](https://mlsanity.com/paper/2511.12044/) (2025)
- [ADEPT: Hierarchical Bayes Approach to Personalized Federated Unsupervised Learning](https://mlsanity.com/paper/2402.12537/) (2024)
- [PDFed: Privacy-Preserving and Decentralized Asynchronous Federated Learning for Diffusion Models](https://mlsanity.com/paper/2409.18245/) (2024)
- [FedPhD: Federated Pruning with Hierarchical Learning of Diffusion Models](https://mlsanity.com/paper/2507.06449/) (2025)
- [FedTSA: A Cluster-based Two-Stage Aggregation Method for Model-heterogeneous Federated Learning](https://mlsanity.com/paper/2407.05098/) (2024)
- [Compositional Generative Modeling from Decentralized Data](https://mlsanity.com/paper/2606.10153/) (2026)

## Abstract (arXiv)

Diffusion models have shown great potential for vision-related tasks, particularly for image generation. However, their training is typically conducted in a centralized manner, relying on data collected from publicly available sources. This approach may not be feasible or practical in many domains, such as the medical field, which involves privacy concerns over data collection. Despite the challenges associated with privacy-sensitive data, such domains could still benefit from valuable vision services provided by diffusion models. Federated learning (FL) plays a crucial role in enabling decentralized model training without compromising data privacy. Instead of collecting data, an FL system gathers model parameters, effectively safeguarding the private data of different parties involved. This makes FL systems vital for managing decentralized learning tasks, especially in scenarios where privacy-sensitive data is distributed across a network of clients. Nonetheless, FL presents its own set of challenges due to its distributed nature and privacy-preserving properties. Therefore, in this study, we explore the FL strategy to train diffusion models, paving the way for the development of federated diffusion models. We conduct experiments on various FL scenarios, and our findings demonstrate that federated diffusion models have great potential to deliver vision services to privacy-sensitive domains.