mlsanityData to 7 Oct 2026

Stable Diffusion-based Data Augmentation for Federated Learning with Non-IID Data

arXiv 2405.07925 ↗First posted 13 May 2024cs.LG cs.AI cs.DCcited by 10

On the map: Diffusion models → Robustness, privacy and adversarial aspects

Abstract (arXiv)

The proliferation of edge devices has brought Federated Learning (FL) to the forefront as a promising paradigm for decentralized and collaborative model training while preserving the privacy of clients' data. However, FL struggles with a significant performance reduction and poor convergence when confronted with Non-Independent and Identically Distributed (Non-IID) data distributions among participating clients. While previous efforts, such as client drift mitigation and advanced server-side model fusion techniques, have shown some success in addressing this challenge, they often overlook the root cause of the performance reduction - the absence of identical data accurately mirroring the global data distribution among clients. In this paper, we introduce Gen-FedSD, a novel approach that harnesses the powerful capability of state-of-the-art text-to-image foundation models to bridge the significant Non-IID performance gaps in FL. In Gen-FedSD, each client constructs textual prompts for each class label and leverages an off-the-shelf state-of-the-art pre-trained Stable Diffusion model to synthesize high-quality data samples. The generated synthetic data is tailored to each client's unique local data gaps and distribution disparities, effectively making the final augmented local data IID. Through extensive experimentation, we demonstrate that Gen-FedSD achieves state-of-the-art performance and significant communication cost savings across various datasets and Non-IID settings.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models. Builds on 48 of its 63 references that are in the map. First posted on arXiv 13 May 2024.

Builds on

48 of 63 references.

References · 48

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  8. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  9. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  10. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  11. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  12. 2018Federated Learning with Non-IID Datacited by 1,099
36 more
  1. 2020Adaptive Federated Optimizationcited by 1,036
  2. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  3. 2021Model-Contrastive Federated Learningcited by 762
  4. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  5. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  6. 2020Federated Learning with Matched Averagingcited by 564
  7. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 537
  8. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  9. 2021SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Modelscited by 415
  10. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
  11. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  12. 2020Adaptive Personalized Federated Learningcited by 353
  13. 2021Federated Learning on Non-IID Data: A Surveycited by 345
  14. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 326
  15. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  16. 2023Effective Data Augmentation With Diffusion Modelscited by 282
  17. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 261
  18. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 242
  19. 2022A Survey on Generative Diffusion Modelcited by 238
  20. 2019Model Fusion via Optimal Transportcited by 217
  21. 2019On the Convergence of Local Descent Methods in Federated Learningcited by 150
  22. 2020Survey of Personalization Techniques for Federated Learningcited by 144
  23. 2023Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generationcited by 142
  24. 2020Lower Bounds and Optimal Algorithms for Personalized Federated Learningcited by 122
  25. 2020Distilled One-Shot Federated Learningcited by 108
  26. 2021FedMix: Approximation of Mixup under Mean Augmented Federated Learningcited by 108
  27. 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
  28. 2023Diversity is Definitely Needed: Improving Model-Agnostic Zero-shot Classification via Stable Diffusioncited by 81
  29. 2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 67
  30. 2020XOR Mixup: Privacy-Preserving Data Augmentation for One-Shot Federated Learningcited by 58
  31. 2020Federated Learning via Synthetic Datacited by 53
  32. 2019Federated Learning Of Out-Of-Vocabulary Wordscited by 51
  33. 2019Energy Demand Prediction with Federated Learning for Electric Vehicle Networkscited by 46
  34. 2022Efficient Distribution Similarity Identification in Clustered Federated Learning via Principal Angles Between Client Data Subspacescited by 36
  35. 2022FedSynth: Gradient Compression via Synthetic Data in Federated Learningcited by 30
  36. 2023GPT-FL: Generative Pre-trained Model-Assisted Federated Learningcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 5.

10 papers.

  1. 2024Federated Learning Client Pruning for Noisy Labelscited by 4
  2. 2025Federated Deconfounding and Debiasing Learning for Out-of-Distribution Generalizationcited by 4
  3. 2025FedDAG: Federated Domain Adversarial Generation Towards Generalizable Medical Image Analysiscited by 2
  4. 2025AIGC-assisted Federated Learning for Edge Intelligence: Architecture Design, Research Challenges and Future Directionscited by 0
  5. 2025Avoid Forgetting by Preserving Global Knowledge Gradients in Federated Learning with Non-IID Datacited by 0
  6. 2026Geometric Knowledge-Assisted Federated Dual Knowledge Distillation Approach Towards Remote Sensing Satellite Imagerycited by 0
  7. 2026FedOT: Ownership Verification and Leakage Tracing via Watermarks for Federated LDMscited by 0
  8. 2026Generative AI and Federated Learning for Intrusion Detection Systems: A Surveycited by 0
  9. 2026WHERE to Generate Matters: Budget-Aware Synthetic Augmentation for Label Skewed Federated Learningcited by 0
  10. 2026FedASAP: Activation Statistics-driven Structured Adaptive Pruning for Efficient Personalized Federated Learning for Lesion Segmentation on brain MRIcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FedDifRC: Unlocking the Potential of Text-to-Image Diffusion Models in Heterogeneous Federated Learning
  2. 2025Federated Learning for Diffusion Models
  3. 2025Diffusion Model-Based Data Synthesis Aided Federated Semi-Supervised Learning
  4. 2025FedPhD: Federated Pruning with Hierarchical Learning of Diffusion Models
  5. 2024Synthetic Data Aided Federated Learning Using Foundation Models
  6. 2025Personalized Federated Training of Diffusion Models with Privacy Guarantees
  7. 2025Parameter-Efficient and Personalized Federated Training of Generative Models at the Edge
  8. 2024Training Diffusion Models with Federated Learning