mlsanityData to 14 Oct 2026

Advances and Open Challenges in Federated Foundation Models

arXiv 2404.15381 ↗First posted 23 Apr 2024cs.LG cs.AIcited by 36

Not on the map. Why?

Abstract (arXiv)

The integration of Foundation Models (FMs) with Federated Learning (FL) presents a transformative paradigm in Artificial Intelligence (AI). This integration offers enhanced capabilities, while addressing concerns of privacy, data decentralization and computational efficiency. This paper provides a comprehensive survey of the emerging field of Federated Foundation Models (FedFM), elucidating their synergistic relationship and exploring novel methodologies, challenges, and future directions that the FL research field needs to focus on in order to thrive in the age of FMs. A systematic multi-tiered taxonomy is proposed, categorizing existing FedFM approaches for model training, aggregation, trustworthiness, and incentivization. Key challenges, including how to enable FL to deal with high complexity of computational demands, privacy considerations, contribution evaluation, and communication efficiency, are thoroughly discussed. Moreover, this paper explores the intricate challenges of communication, scalability and security inherent in training/fine-tuning FMs via FL. It highlights the potential of quantum computing to revolutionize the processes of training, inference, optimization and security. This survey also introduces the implementation requirement of FedFM and some practical FedFM applications. It highlights lessons learned with a clear understanding of our findings for FedFM. Finally, this survey not only provides insights into the current state and challenges of FedFM, but also offers a blueprint for future research directions, emphasizing the need for developing trustworthy solutions. It serves as a foundational guide for researchers and practitioners interested in contributing to this interdisciplinary and rapidly advancing field.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 195 of its 323 references that are in the map. First posted on arXiv 23 Apr 2024.

Builds on

195 of 323 references.

References · 195

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2023GPT-4 Technical Reportcited by 18,396
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  10. 2023Segment Anythingcited by 8,234
  11. 2023Visual Instruction Tuningcited by 7,869
  12. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
38 more
  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2019Advances and Open Problems in Federated Learningcited by 3,046
  9. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  10. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  12. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  13. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  14. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  15. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  16. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,106
  17. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  18. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,055
  19. 2019Deep Leakage from Gradientscited by 1,021
  20. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  21. 2022A Generalist Agentcited by 808
  22. 2018How To Backdoor Federated Learningcited by 765
  23. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  24. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 711
  25. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  26. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  27. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 564
  28. 2020Federated Learning with Matched Averagingcited by 564
  29. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 519
  30. 2020An Efficient Framework for Clustered Federated Learningcited by 511
  31. 2019Local Model Poisoning Attacks to Byzantine-Robust Federated Learningcited by 493
  32. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
  33. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 465
  34. 2021Towards Personalized Federated Learningcited by 464
  35. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  36. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 455
  37. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  38. 2020iDLG: Improved Deep Leakage from Gradientscited by 382

50 most cited of 195 All 195 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 19; 2026: 16.

36 papers.

Sort:

By area:

  1. 2025Can Textual Gradient Work in Federated Learning?cited by 15
  2. 2025NoT: Federated Unlearning via Weight Negationcited by 15
  3. 2025Quantum Federated Learning: A Comprehensive Surveycited by 10
  4. 2025Multi-Modal Multi-Task (M3T) Federated Foundation Models for Embodied AI: Potentials and Challenges for Edge Integrationcited by 6
  5. 2025Federated Deconfounding and Debiasing Learning for Out-of-Distribution Generalizationcited by 4
  6. 2025A Survey on Foundation Models for Personalized Federated Intelligencecited by 3
  7. 2025Technical and Legal Aspects of Federated Learning in Bioinformatics: Applications, Challenges and Opportunitiescited by 2
  8. 2025Blockchain-Enabled Privacy-Preserving Second-Order Federated Edge Learning in Personalized Healthcarecited by 2
  9. 2025APFL: Analytic Personalized Federated Learning via Dual-Stream Least Squarescited by 2
  10. 2025Hierarchical Federated Foundation Models over Wireless Networks for Multi-Modal Multi-Task Intelligence: Integration of Edge Learning with D2D/P2P-Enabled Fog Learning Architecturescited by 2
  11. 2025Global Prompt Refinement with Non-Interfering Attention Masking for One-Shot Federated Learningcited by 2
  12. 2024Federated Vision-Language-Recommendation with Personalized Fusioncited by 1
24 more
  1. 2026Toward Federated Large Language Models in Medicine: A Parameter-Efficient Framework for Privacy-Preserving, Multi-Institutional Adaptationcited by 1
  2. 2026DeepAFL: Deep Analytic Federated Learningcited by 1
  3. 2026Stabilized Fine-Tuning with LoRA in Federated Learning: Mitigating the Side Effect of Client Size and Rank via the Scaling Factorcited by 1
  4. 2025Local Data Quantity-Aware Weighted Averaging for Federated Learning with Dishonest Clientscited by 0
  5. 2025Voronoi-grid-based Pareto Front Learning and Its Application to Collaborative Federated Learningcited by 0
  6. 2025A Survey of Personalized Federated Foundation Models for Privacy-Preserving Recommendationcited by 0
  7. 2025DAG-AFL:Directed Acyclic Graph-based Asynchronous Federated Learningcited by 0
  8. 2025Large-Small Model Collaborative Framework for Federated Continual Learningcited by 0
  9. 2025Synergies between Federated Foundation Models and Smart Power Gridscited by 0
  10. 2025FRIEREN: Federated Learning with Vision-Language Regularization for Segmentationcited by 0
  11. 2025OvA-LP: A Simple and Efficient Framework for Federated Learning on Non-IID Datacited by 0
  12. 2026Res-MIA: A Training-Free Resolution-Based Membership Inference Attack on Federated Learning Modelscited by 0
  13. 2026FedGRPO: Privately Optimizing Foundation Models with Group-Relative Rewards from Domain Clientcited by 0
  14. 2026HiLoRA: Hierarchical Low-Rank Adaptation for Personalized Federated Learningcited by 0
  15. 2026FedSDR: Federated Self-Distillation with Rectificationcited by 0
  16. 2026Silent Failures in Federated Personalization of Foundation Modelscited by 0
  17. 2026Federated Foundation Models over Vehicular Networkscited by 0
  18. 2026Quantization in Federated Learning: Methods, Challenges and Future Directionscited by 0
  19. 2026Evaluating Federated Pre-Training: On the Reliability of Downstream Fine-Tuning and Intrinsic Evaluationcited by 0
  20. 2026EFFEKT: Efficient Federated Knowledge Transfer to Foundation Modelscited by 0
  21. 2026Task-oriented Framework for Communication-Efficient Federated Learning: From Isolated Optimization to Holistic Synergycited by 0
  22. 2026FairMean: Promoting Fairness in Distributed Learning under Label Poisoning Attackscited by 0
  23. 2026A generalizable structural brain MRI foundation model built through dual-priority federated pretrainingcited by 0
  24. 2026Byzantine-Resilient Federated Large Language Model via Hypergraph Message Passingcited by 0