# Synergizing Foundation Models and Federated Learning: A Survey

arXiv: https://arxiv.org/abs/2406.12844 · HTML: https://mlsanity.com/paper/2406.12844/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 141 of its 231 references that are in the map.
- First posted on arXiv 18 Jun 2024.

## Builds on (most cited first, 30 of 141)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)

## Built on by (most cited first, 14 of 14)

- [Ten Challenging Problems in Federated Foundation Models](https://mlsanity.com/paper/2502.12176/) (2025)
- [A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learning](https://mlsanity.com/paper/2504.21099/) (2025)
- [Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions](https://mlsanity.com/paper/2503.16585/) (2025)
- [Vision Foundation Models in Medical Image Analysis: Advances and Challenges](https://mlsanity.com/paper/2502.14584/) (2025)
- [Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stages](https://mlsanity.com/paper/2603.12658/) (2026)
- [PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2411.19335/) (2024)
- [A Survey on Foundation Models for Personalized Federated Intelligence](https://mlsanity.com/paper/2505.06907/) (2025)
- [Federated Learning from Molecules to Processes: A Perspective](https://mlsanity.com/paper/2506.18525/) (2025)
- [Simple Yet Effective: Extracting Private Data Across Clients in Federated Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2506.06060/) (2025)
- [Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights](https://mlsanity.com/paper/2509.05142/) (2025)
- [Integration of TinyML and LargeML: A Survey of 6G and Beyond](https://mlsanity.com/paper/2505.15854/) (2025)
- [Improving Learning of New Diseases through Knowledge-Enhanced Initialization for Federated Adapter Tuning](https://mlsanity.com/paper/2508.10299/) (2025)
- [Federated Medical Image Classification under Class and Domain Imbalance exploiting Synthetic Sample Generation](https://mlsanity.com/paper/2604.26324/) (2026)
- [Evaluating Federated Pre-Training: On the Reliability of Downstream Fine-Tuning and Intrinsic Evaluation](https://mlsanity.com/paper/2607.28658/) (2026)

## Abstract (arXiv)

Over the past few years, the landscape of Artificial Intelligence (AI) has been reshaped by the emergence of Foundation Models (FMs). Pre-trained on massive datasets, these models exhibit exceptional performance across diverse downstream tasks through adaptation techniques like fine-tuning and prompt learning. More recently, the synergy of FMs and Federated Learning (FL) has emerged as a promising paradigm, often termed Federated Foundation Models (FedFM), allowing for collaborative model adaptation while preserving data privacy. This survey paper provides a systematic review of the current state of the art in FedFM, offering insights and guidance into the evolving landscape. Specifically, we present a comprehensive multi-tiered taxonomy based on three major dimensions, namely efficiency, adaptability, and trustworthiness. To facilitate practical implementation and experimental research, we undertake a thorough review of existing libraries and benchmarks. Furthermore, we discuss the diverse real-world applications of this paradigm across multiple domains. Finally, we outline promising research directions to foster future advancements in FedFM. Overall, this survey serves as a resource for researchers and practitioners, offering a thorough understanding of FedFM's role in revolutionizing privacy-preserving AI and pointing toward future innovations in this promising area. A periodically updated paper collection on FM-FL is available at https://github.com/lishenghui/awesome-fm-fl.