mlsanityData to 7 Oct 2026

Federated Generative Learning with Foundation Models

arXiv 2306.16064 ↗First posted 28 Jun 2023cs.LG cs.AIcited by 23

Not on the map. Why?

Abstract (arXiv)

Existing approaches in Federated Learning (FL) mainly focus on sending model parameters or gradients from clients to a server. However, these methods are plagued by significant inefficiency, privacy, and security concerns. Thanks to the emerging foundation generative models, we propose a novel federated learning framework, namely Federated Generative Learning. In this framework, each client can create text embeddings that are tailored to their local data, and send embeddings to the server. Then the informative training data can be synthesized remotely on the server using foundation generative models with these embeddings, which can benefit FL tasks. Our proposed framework offers several advantages, including increased communication efficiency, robustness to data heterogeneity, substantial performance improvements, and enhanced privacy protection. We validate these benefits through extensive experiments conducted on 12 datasets. For example, on the ImageNet100 dataset with a highly skewed data distribution, our method outperforms FedAvg by 12% in a single communication round, compared to FedAvg's performance over 200 communication rounds. We have released the code for all experiments conducted in this study.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 77 of its 91 references that are in the map. First posted on arXiv 28 Jun 2023.

Builds on

77 of 91 references.

References · 77

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022Scalable Diffusion Models with Transformerscited by 5,931
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  4. 2019Advances and Open Problems in Federated Learningcited by 3,021
  5. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  6. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  7. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  8. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  9. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  10. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  11. 2018Federated Learning with Non-IID Datacited by 1,099
  12. 2020Adaptive Federated Optimizationcited by 1,036
  13. 2019Deep Leakage from Gradientscited by 1,004
  14. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  15. 2021Membership Inference Attacks From First Principlescited by 839
  16. 2019Towards Federated Learning at Scale: System Designcited by 826
  17. 2021Model-Contrastive Federated Learningcited by 762
  18. 2023Extracting Training Data from Diffusion Modelscited by 723
  19. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 682
  20. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  21. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  22. 2018ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Modelscited by 475
  23. 2022Dataset Distillation by Matching Training Trajectoriescited by 437
  24. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 418
  25. 2020iDLG: Improved Deep Leakage from Gradientscited by 367
  26. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  27. 2023Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Modelscited by 297
  28. 2021FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Spacecited by 284
  29. 2020Systematic Evaluation of Privacy Risks of Machine Learning Modelscited by 273
  30. 2019White-box vs Black-box: Bayes Optimal Strategies for Membership Inferencecited by 265
  31. 2023Understanding and Mitigating Copying in Diffusion Modelscited by 205
  32. 2020Can AI help in screening Viral and COVID-19 pneumonia?cited by 200
  33. 2021Federated Learning for Open Bankingcited by 138
  34. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 133
  35. 2019One-Shot Federated Learningcited by 124
  36. 2020Distilled One-Shot Federated Learningcited by 108
  37. 2022Federated Learning with Label Distribution Skew via Logits Calibrationcited by 108
  38. 2021DENSE: Data-Free One-Shot Federated Learningcited by 100

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 10; 2025: 8; 2026: 2.

23 papers.

Sort:

By area:

  1. 2024Jailbreaking Prompt Attack: A Controllable Adversarial Attack against Diffusion Modelscited by 68
  2. 2023Real-Fake: Effective Training Data Synthesis Through Distribution Matchingcited by 35
  3. 2024DiffClass: Diffusion-Based Class Incremental Learningcited by 23
  4. 2024Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learningcited by 23
  5. 2024FedDEO: Description-Enhanced One-Shot Federated Learning with Diffusion Modelscited by 17
  6. 2024FedBiP: Heterogeneous One-Shot Federated Learning with Personalized Latent Diffusion Modelscited by 16
  7. 2025Data-free Knowledge Distillation with Diffusion Modelscited by 16
  8. 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
  9. 2024One-shot Federated Learning via Synthetic Distiller-Distillate Communicationcited by 12
  10. 2025Does Training with Synthetic Data Truly Protect Privacy?cited by 12
  11. 2025Vision Foundation Models in Medical Image Analysis: Advances and Challengescited by 9
  12. 2025One-Shot Federated Learning with Classifier-Free Diffusion Modelscited by 7
11 more
  1. 2025Federated Learning for Medical Image Classification: A Comprehensive Benchmarkcited by 6
  2. 2023One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Modelscited by 5
  3. 2023FedMS: Federated Learning with Mixture of Sparsely Activated Foundations Modelscited by 4
  4. 2024Harnessing Federated Generative Learning for Green and Sustainable Internet of Thingscited by 1
  5. 2025A New One-Shot Federated Learning Framework for Medical Imaging Classification with Feature-Guided Rectified Flow and Knowledge Distillationcited by 1
  6. 2024Adaptive Frequency Enhancement Network for Single Image Derainingcited by 0
  7. 2024Recovering Global Data Distribution Locally in Federated Learningcited by 0
  8. 2025WarmFed: Federated Learning with Warm-Start for Globalization and Personalization Via Personalized Diffusion Modelscited by 0
  9. 2025FedMP: Tackling Medical Feature Heterogeneity in Federated Learning from a Manifold Perspectivecited by 0
  10. 2026Feature-Aware One-Shot Federated Learning via Hierarchical Token Sequencescited by 0
  11. 2026CRIP: Channel Level Representation Injection for Personalized One-Shot Federated Learningcited by 0