mlsanityData to 14 Oct 2026

Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Models

arXiv 2305.11414 ↗First posted 19 May 2023cs.LG cs.AI cs.CRcited by 50

Not on the map. Why?

Abstract (arXiv)

Foundation Models (FMs), such as LLaMA, BERT, GPT, ViT, and CLIP, have demonstrated remarkable success in a wide range of applications, driven by their ability to leverage vast amounts of data for pre-training. However, optimizing FMs often requires access to sensitive data, raising privacy concerns and limiting their applicability in many domains. In this paper, we propose the Federated Foundation Models (FFMs) paradigm, which combines the benefits of FMs and Federated Learning (FL) to enable privacy-preserving and collaborative learning across multiple end-users. We discuss the potential benefits and challenges of integrating FL into the lifespan of FMs, covering pre-training, fine-tuning, and application. We further outline potential future research avenues in FFM, including FFM pre-training, FFM fine-tuning, and federated prompt tuning, which allow the development of more personalized and context-aware models while ensuring data privacy. Moreover, we explore the possibility of continual/lifelong learning in FFMs, as increased computational power at the edge may unlock the potential for optimizing FMs using newly generated private data close to the data source. The proposed FFM concepts offer a flexible and scalable framework for training large language models in a privacy-preserving manner, setting the stage for subsequent advancements in both FM training and federated learning.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Builds on 30 of its 56 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

30 of 56 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
18 more
  1. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  2. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  3. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  4. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  5. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  6. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  7. 2021MetaICL: Learning to Learn In Contextcited by 454
  8. 2020Adaptive Personalized Federated Learningcited by 358
  9. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  10. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 222
  11. 2019Accelerating Federated Learning via Momentum Gradient Descentcited by 76
  12. 2021Topology-Aware Network Pruning using Multi-stage Graph Embedding and Reinforcement Learningcited by 30
  13. 2020Auto Graph Encoder-Decoder for Neural Network Pruningcited by 26
  14. 2022AsyncFedED: Asynchronous Federated Learning with Euclidean Distance based Adaptive Weight Aggregationcited by 19
  15. 2021Differential Privacy Meets Federated Learning under Communication Constraintscited by 16
  16. 2021SPATL: Salient Parameter Aggregation and Transfer Learning for Heterogeneous Clients in Federated Learningcited by 11
  17. 2022Enhancing Heterogeneous Federated Learning with Knowledge Extraction and Multi-Model Fusioncited by 10
  18. 2022Study of the performance and scalability of federated learning for medical imaging with intermittent clientscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 18; 2025: 18; 2026: 6.

50 papers.

Sort:

By area:

  1. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
  2. 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 84
  3. 2023FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learningcited by 60
  4. 2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 55
  5. 2024Advances and Open Challenges in Federated Foundation Modelscited by 36
  6. 2023Integration of Large Language Models and Federated Learningcited by 31
  7. 2024The Future of Large Language Model Pre-training is Federatedcited by 30
  8. 2024A Survey on Efficient Federated Learning Methods for Foundation Model Trainingcited by 29
  9. 2024Federated Large Language Models: Current Progress and Future Directionscited by 27
  10. 2023Federated Generative Learning with Foundation Modelscited by 23
  11. 2025Ten Challenging Problems in Federated Foundation Modelscited by 20
  12. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 17
38 more
  1. 2024FedPFT: Federated Proxy Fine-Tuning of Foundation Modelscited by 16
  2. 2024Federated Learning in Practice: Reflections and Projectionscited by 15
  3. 2024Vertical Federated Learning for Effectiveness, Security, Applicability: A Surveycited by 14
  4. 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
  5. 2023Building Privacy-Preserving and Secure Geospatial Artificial Intelligence Foundation Modelscited by 12
  6. 2024Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcarecited by 12
  7. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  8. 2023Bridging the Gap Between Foundation Models and Heterogeneous Federated Learningcited by 9
  9. 2024Analysis of Privacy Leakage in Federated Large Language Modelscited by 9
  10. 2025Federated Low-Rank Adaptation for Foundation Models: A Surveycited by 9
  11. 2023FedFN: Feature Normalization for Alleviating Data Heterogeneity Problem in Federated Learningcited by 8
  12. 2025FedMentalCare: Towards Privacy-Preserving Fine-Tuned LLMs to Analyze Mental Health Status Using Federated Learning Frameworkcited by 8
  13. 2025One-Shot Federated Learning with Classifier-Free Diffusion Modelscited by 7
  14. 2025Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocationcited by 7
  15. 2024FedPIA -- Permuting and Integrating Adapters leveraging Wasserstein Barycenters for Finetuning Foundation Models in Multi-Modal Federated Learningcited by 6
  16. 2023FDAPT: Federated Domain-adaptive Pre-training for Language Modelscited by 5
  17. 2024F3OCUS -- Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristicscited by 4
  18. 2025A General Pseudonymization Framework for Cloud-Based LLMs: Replacing Privacy Information in Controlled Text Generationcited by 3
  19. 2025A Survey on Foundation Models for Personalized Federated Intelligencecited by 3
  20. 2025Personalized Federated Learning via Dual-Prompt Optimization and Cross Fusioncited by 3
  21. 2026FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuningcited by 3
  22. 2024Metarobotics for Industry and Society: Vision, Technologies, and Opportunitiescited by 2
  23. 2025Federated Adapter on Foundation Models: An Out-Of-Distribution Approachcited by 2
  24. 2024Heterogeneous Federated Learning with Splited Language Modelcited by 1
  25. 2025Zero-Trust Foundation Models: A New Paradigm for Secure and Collaborative Artificial Intelligence for Internet of Thingscited by 1
  26. 2025FedPhD: Federated Pruning with Hierarchical Learning of Diffusion Modelscited by 1
  27. 2025Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insightscited by 1
  28. 2025CollaPipe: Adaptive Segment-Optimized Pipeline Parallelism for Collaborative LLM Training in Heterogeneous Edge Networkscited by 1
  29. 2026ERIS: Enhancing the Privacy-Utility-Scalability Trade-Off in Federated Learning via Federated Shard Aggregationcited by 1
  30. 2024Navigating the Future of Federated Recommendation Systems with Foundation Modelscited by 0
  31. 2024Federated Knowledge Transfer Fine-tuning Large Server Model with Resource-Constrained IoT Clientscited by 0
  32. 2025A Survey of Personalized Federated Foundation Models for Privacy-Preserving Recommendationcited by 0
  33. 2025On the Evolution of Federated Post-Training Large Language Models: A Model Accessibility Viewcited by 0
  34. 2025Bi-level Personalization for Federated Foundation Models: A Task-vector Aggregation Approachcited by 0
  35. 2026AviationLMM: A Large Multimodal Foundation Model for Civil Aviationcited by 0
  36. 2026Unextractable Protocol Models: Collaborative Training and Inference without Weight Materializationcited by 0
  37. 2026Amortizing Federated Adaptation: Hypernetwork Driven LoRA for Personalized Foundation Modelscited by 0
  38. 2026Global Federated Learning Strategies for Building Efficient Personalized Modelscited by 0