mlsanityData to 14 Oct 2026

Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights

arXiv 2509.05142 ↗First posted 5 Sep 2025cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Federated learning has the potential to unlock siloed data and distributed resources by enabling collaborative model training without sharing private data. As more complex foundational models gain widespread use, the need to expand training resources and integrate privately owned data grows as well. In this article, we explore the intersection of federated learning and foundational models, aiming to identify, categorize, and characterize technical methods that integrate the two paradigms. As a unified survey is currently unavailable, we present a literature survey structured around a novel taxonomy that follows the development life-cycle stages, along with a technical comparison of available methods. Additionally, we provide practical insights and guidelines for implementing and evolving these methods, with a specific focus on the healthcare domain as a case study, where the potential impact of federated learning and foundational models is considered significant. Our survey covers multiple intersecting topics, including but not limited to federated learning, self-supervised learning, fine-tuning, distillation, and transfer learning. Initially, we retrieved and reviewed a set of over 4,200 articles. This collection was narrowed to more than 250 thoroughly reviewed articles through inclusion criteria, featuring 42 unique methods. The methods were used to construct the taxonomy and enabled their comparison based on complexity, efficiency, and scalability. We present these results as a self-contained overview that not only summarizes the state of the field but also provides insights into the practical aspects of adopting, evolving, and integrating foundational models with federated learning.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 81 of its 107 references that are in the map. First posted on arXiv 5 Sep 2025.

Builds on

81 of 107 references.

References · 81

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2023Segment Anythingcited by 8,234
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  11. 2019Advances and Open Problems in Federated Learningcited by 3,046
  12. 2020Knowledge Distillation: A Surveycited by 1,603
38 more
  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  2. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  3. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  4. 2021Model-Contrastive Federated Learningcited by 766
  5. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  6. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 222
  7. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 187
  8. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
  9. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  10. 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 142
  11. 2023On the Challenges and Perspectives of Foundation Models for Medical Image Analysiscited by 135
  12. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 134
  13. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 123
  14. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
  15. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 110
  16. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 108
  17. 2021DENSE: Data-Free One-Shot Federated Learningcited by 100
  18. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 99
  19. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 93
  20. 2022FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learningcited by 91
  21. 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 84
  22. 2021Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learningcited by 78
  23. 2023Federated Learning for Medical Image Analysis: A Surveycited by 78
  24. 2023Foundational Models in Medical Imaging: A Comprehensive Survey and Future Visioncited by 76
  25. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 75
  26. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 74
  27. 2022Divergence-aware Federated Self-Supervised Learningcited by 73
  28. 2021Collaborative Unsupervised Visual Representation Learning from Decentralized Datacited by 69
  29. 2023FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learningcited by 60
  30. 2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 55
  31. 2023Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generationcited by 54
  32. 2023Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Modelscited by 50
  33. 2023Recent Advances on Federated Learning: A Systematic Surveycited by 41
  34. 2023InfoNCE Loss Provably Learns Cluster-Preserving Representationscited by 37
  35. 2021Emerging Trends in Federated Learning: From Model Fusion to Federated X Learningcited by 35
  36. 2022FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformerscited by 33
  37. 2023L-DAWA: Layer-wise Divergence Aware Weight Aggregation in Federated Self-Supervised Visual Representation Learningcited by 30
  38. 2024The Future of Large Language Model Pre-training is Federatedcited by 30

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Beyond Weights and Gradients: A Taxonomy of Federated Learning Messagescited by 0