mlsanityData to 7 Oct 2026

Synergizing Foundation Models and Federated Learning: A Survey

arXiv 2406.12844 ↗First posted 18 Jun 2024cs.LG cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

Over the past few years, the landscape of Artificial Intelligence (AI) has been reshaped by the emergence of Foundation Models (FMs). Pre-trained on massive datasets, these models exhibit exceptional performance across diverse downstream tasks through adaptation techniques like fine-tuning and prompt learning. More recently, the synergy of FMs and Federated Learning (FL) has emerged as a promising paradigm, often termed Federated Foundation Models (FedFM), allowing for collaborative model adaptation while preserving data privacy. This survey paper provides a systematic review of the current state of the art in FedFM, offering insights and guidance into the evolving landscape. Specifically, we present a comprehensive multi-tiered taxonomy based on three major dimensions, namely efficiency, adaptability, and trustworthiness. To facilitate practical implementation and experimental research, we undertake a thorough review of existing libraries and benchmarks. Furthermore, we discuss the diverse real-world applications of this paradigm across multiple domains. Finally, we outline promising research directions to foster future advancements in FedFM. Overall, this survey serves as a resource for researchers and practitioners, offering a thorough understanding of FedFM's role in revolutionizing privacy-preserving AI and pointing toward future innovations in this promising area. A periodically updated paper collection on FM-FL is available at https://github.com/lishenghui/awesome-fm-fl.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 141 of its 231 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

141 of 231 references.

References · 141

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  10. 2023Segment Anythingcited by 8,158
  11. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
38 more
  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2019Advances and Open Problems in Federated Learningcited by 3,021
  10. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  12. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  13. 2022Visual Prompt Tuningcited by 1,674
  14. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  15. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  16. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  17. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  18. 2020Adaptive Federated Optimizationcited by 1,036
  19. 2019Deep Leakage from Gradientscited by 1,004
  20. 2016HyperNetworkscited by 1,003
  21. 2018How To Backdoor Federated Learningcited by 745
  22. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 707
  23. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 698
  24. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  25. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  26. 2019How multilingual is Multilingual BERT?cited by 642
  27. 2020The Future of Digital Health with Federated Learningcited by 625
  28. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  29. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  30. 2019Federated Learning with Differential Privacy: Algorithms and Performance Analysiscited by 528
  31. 2019Local Model Poisoning Attacks to Byzantine-Robust Federated Learningcited by 479
  32. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  33. 2023A Survey on Large Language Models for Recommendationcited by 378
  34. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 349
  35. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  36. 2018Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooringcited by 334
  37. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
  38. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333

50 most cited of 141 All 141 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 3.

14 papers.

Sort:

By area:

  1. 2025Ten Challenging Problems in Federated Foundation Modelscited by 20
  2. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 15
  3. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  4. 2025Vision Foundation Models in Medical Image Analysis: Advances and Challengescited by 9
  5. 2024PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuningcited by 5
  6. 2026Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stagescited by 4
  7. 2025A Survey on Foundation Models for Personalized Federated Intelligencecited by 2
  8. 2025Federated Learning from Molecules to Processes: A Perspectivecited by 2
  9. 2025Simple Yet Effective: Extracting Private Data Across Clients in Federated Fine-Tuning of Large Language Modelscited by 1
  10. 2025Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insightscited by 1
  11. 2025Integration of TinyML and LargeML: A Survey of 6G and Beyondcited by 0
  12. 2025Improving Learning of New Diseases through Knowledge-Enhanced Initialization for Federated Adapter Tuningcited by 0
2 more
  1. 2026Federated Medical Image Classification under Class and Domain Imbalance exploiting Synthetic Sample Generationcited by 0
  2. 2026Evaluating Federated Pre-Training: On the Reliability of Downstream Fine-Tuning and Intrinsic Evaluationcited by 0