# When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directions

arXiv: https://arxiv.org/abs/2306.15546 · HTML: https://mlsanity.com/paper/2306.15546/

## Key facts

- Cited by 84 papers in the map (data through 14 Oct 2026).
- Builds on 112 of its 183 references that are in the map.
- First posted on arXiv 27 Jun 2023.

## Builds on (most cited first, 30 of 112)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)

## Built on by (most cited first, 30 of 84)

- [Recommender Systems in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2307.02046/) (2023)
- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directions](https://mlsanity.com/paper/2404.03264/) (2024)
- [FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning](https://mlsanity.com/paper/2308.12305/) (2023)
- [A Comprehensive Survey of Foundation Models in Medicine](https://mlsanity.com/paper/2406.10729/) (2024)
- [Dual-Personalizing Adapter for Federated Foundation Models](https://mlsanity.com/paper/2403.19211/) (2024)
- [Advances and Open Challenges in Federated Foundation Models](https://mlsanity.com/paper/2404.15381/) (2024)
- [Integration of Large Language Models and Federated Learning](https://mlsanity.com/paper/2307.08925/) (2023)
- [Artificial General Intelligence (AGI)-Native Wireless Systems: A Journey Beyond 6G](https://mlsanity.com/paper/2405.02336/) (2024)
- [Enhancing One-Shot Federated Learning Through Data and Ensemble Co-Boosting](https://mlsanity.com/paper/2402.15070/) (2024)
- [Federated Large Language Models: Current Progress and Future Directions](https://mlsanity.com/paper/2409.15723/) (2024)
- [FwdLLM: Efficient FedLLM using Forward Gradient](https://mlsanity.com/paper/2308.13894/) (2023)
- [Grounding Foundation Models through Federated Transfer Learning: A General Framework](https://mlsanity.com/paper/2311.17431/) (2023)
- [Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Models](https://mlsanity.com/paper/2402.01749/) (2024)
- [Ten Challenging Problems in Federated Foundation Models](https://mlsanity.com/paper/2502.12176/) (2025)
- [Advances in APPFL: A Comprehensive and Extensible Federated Learning Framework](https://mlsanity.com/paper/2409.11585/) (2024)
- [An Upload-Efficient Scheme for Transferring Knowledge From a Server-Side Pre-trained Generator to Clients in Heterogeneous Federated Learning](https://mlsanity.com/paper/2403.15760/) (2024)
- [A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learning](https://mlsanity.com/paper/2504.21099/) (2025)
- [FedPFT: Federated Proxy Fine-Tuning of Foundation Models](https://mlsanity.com/paper/2404.11536/) (2024)
- [Federated Learning in Practice: Reflections and Projections](https://mlsanity.com/paper/2410.08892/) (2024)
- [Vertical Federated Learning for Effectiveness, Security, Applicability: A Survey](https://mlsanity.com/paper/2405.17495/) (2024)
- [Parametric Feature Transfer: One-shot Federated Learning with Foundation Models](https://mlsanity.com/paper/2402.01862/) (2024)
- [Federated Foundation Models on Heterogeneous Time Series](https://mlsanity.com/paper/2412.08906/) (2024)
- [Towards Multi-modal Transformers in Federated Learning](https://mlsanity.com/paper/2404.12467/) (2024)
- [Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcare](https://mlsanity.com/paper/2405.06784/) (2024)
- [Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions](https://mlsanity.com/paper/2503.16585/) (2025)
- [FeDaL: Federated Dataset Learning for General Time Series Foundation Models](https://mlsanity.com/paper/2508.04045/) (2025)
- [Bridging the Gap Between Foundation Models and Heterogeneous Federated Learning](https://mlsanity.com/paper/2310.00247/) (2023)
- [ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learning](https://mlsanity.com/paper/2310.05143/) (2023)
- [Vision Foundation Models in Medical Image Analysis: Advances and Challenges](https://mlsanity.com/paper/2502.14584/) (2025)

## Abstract (arXiv)

The intersection of Foundation Model (FM) and Federated Learning (FL) presents a unique opportunity to unlock new possibilities for real-world applications. On the one hand, FL, as a collaborative learning paradigm, help address challenges in FM development by expanding data availability, enabling computation sharing, facilitating the collaborative development of FMs, tackling continuous data update, avoiding FM monopoly, response delay and FM service down. On the other hand, FM, equipped with pre-trained knowledge and exceptional performance, can serve as a robust starting point for FL. It can also generate synthetic data to enrich data diversity and enhance overall performance of FL. Meanwhile, FM unlocks new sharing paradigm and multi-task and multi-modality capabilities for FL. By examining the interplay between FL and FM, this paper presents the motivations, challenges, and future directions of empowering FL with FM and empowering FM with FL. We hope that this work provides a good foundation to inspire future research efforts to drive advancements in both fields.