mlsanityData to 7 Oct 2026

A Survey on Efficient Federated Learning Methods for Foundation Model Training

arXiv 2401.04472 ↗First posted 9 Jan 2024cs.LG cs.AI cs.DCcited by 29

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) has become an established technique to facilitate privacy-preserving collaborative training across a multitude of clients. However, new approaches to FL often discuss their contributions involving small deep-learning models only and focus on training full models on clients. In the wake of Foundation Models (FM), the reality is different for many deep learning applications. Typically, FMs have already been pre-trained across a wide variety of tasks and can be fine-tuned to specific downstream tasks over significantly smaller datasets than required for full model training. However, access to such datasets is often challenging. By its design, FL can help to open data silos. With this survey, we introduce a novel taxonomy focused on computational and communication efficiency, the vital elements to make use of FMs in FL systems. We discuss the benefits and drawbacks of parameter-efficient fine-tuning (PEFT) for FL applications, elaborate on the readiness of FL frameworks to work with FMs, and provide future research opportunities on how to evaluate generative models in FL as well as the interplay of privacy and PEFT.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 53 of its 79 references that are in the map. First posted on arXiv 9 Jan 2024.

Builds on

53 of 79 references.

References · 53

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  8. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  9. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  10. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  11. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 449
  12. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
38 more
  1. 2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 371
  2. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 364
  3. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 349
  4. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
  5. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 298
  6. 2021Federated Learning for Internet of Things: A Comprehensive Surveycited by 247
  7. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 219
  8. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  9. 2021Block Pruning For Faster Transformerscited by 176
  10. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
  11. 2019Federated Adversarial Domain Adaptationcited by 140
  12. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  13. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
  14. 2019Distributed Learning with Compressed Gradient Differencescited by 110
  15. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 92
  16. 2022FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learningcited by 86
  17. 2022NVIDIA FLARE: Federated Learning from Simulation to Real-Worldcited by 78
  18. 2023FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learningcited by 70
  19. 2021Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Frameworkcited by 69
  20. 2017Adaptive Quantization for Deep Neural Networkcited by 67
  21. 2021From Distributed Machine Learning to Federated Learning: A Surveycited by 65
  22. 2023Federated Learning of Gboard Language Models with Differential Privacycited by 64
  23. 2020LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasetscited by 60
  24. 2021OpenFL: An open-source framework for Federated Learningcited by 58
  25. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 58
  26. 2020IBM Federated Learning: an Enterprise Framework White Paper V0.1cited by 57
  27. 2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
  28. 2022FederatedScope: A Flexible Federated Learning Platform for Heterogeneitycited by 53
  29. 2023Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Modelscited by 49
  30. 2022SoteriaFL: A Unified Framework for Private Federated Learning with Communication Compressioncited by 34
  31. 2022Sparse Random Networks for Communication-Efficient Federated Learningcited by 32
  32. 2017A Hybrid Deep Learning Architecture for Privacy-Preserving Mobile Analyticscited by 30
  33. 2022FedOBD: Opportunistic Block Dropout for Efficiently Training Large-scale Neural Networks through Federated Learningcited by 17
  34. 2023FLINT: A Platform for Federated Learning Integrationcited by 17
  35. 2023Federated Fine-Tuning of LLMs on the Very Edge: The Good, the Bad, the Uglycited by 17
  36. 2019Substra: a framework for privacy-preserving, traceable and collaborative Machine Learningcited by 14
  37. 2021H-FL: A Hierarchical Communication-Efficient and Privacy-Protected Architecture for Federated Learningcited by 14
  38. 2022Distributed Pruning Towards Tiny Neural Networks in Federated Learningcited by 11

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 14; 2026: 5.

29 papers.

Sort:

By area:

  1. 2024Advances and Open Challenges in Federated Foundation Modelscited by 33
  2. 2024Federated Large Language Models: Current Progress and Future Directionscited by 22
  3. 2025Ten Challenging Problems in Federated Foundation Modelscited by 20
  4. 2024Federated Learning in Practice: Reflections and Projectionscited by 15
  5. 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
  6. 2024The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resourcescited by 11
  7. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  8. 2025Vision Foundation Models in Medical Image Analysis: Advances and Challengescited by 9
  9. 2025Federated Low-Rank Adaptation for Foundation Models: A Surveycited by 9
  10. 2024COALA: A Practical and Vision-Centric Federated Learning Platformcited by 7
  11. 2025One-shot Federated Learning Methods: A Practical Guidecited by 6
  12. 2024PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuningcited by 5
17 more
  1. 2025Fairness in Federated Learning: Trends, Challenges, and Opportunitiescited by 3
  2. 2025Research in Collaborative Learning Does Not Serve Cross-Silo Federated Learning in Practicecited by 3
  3. 2025A Survey on Foundation Models for Personalized Federated Intelligencecited by 2
  4. 2024Federated Learning and AI Regulation in the European Union: Who is Responsible? -- An Interdisciplinary Analysiscited by 1
  5. 2025Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insightscited by 1
  6. 2026When More Parameters Hurt: Foundation Model Priors Amplify Worst-Client Disparity Under Extreme Federated Heterogeneitycited by 1
  7. 2024Consensus learning: A novel decentralised ensemble learning paradigmcited by 0
  8. 2024Navigating the Future of Federated Recommendation Systems with Foundation Modelscited by 0
  9. 2025Federated Foundation Language Model Post-Training Should Focus on Open-Source Modelscited by 0
  10. 2025Overcoming Challenges of Partial Client Participation in Federated Learning : A Comprehensive Reviewcited by 0
  11. 2025FedDPG: An Adaptive Yet Efficient Prompt-tuning Approach in Federated Learning Settingscited by 0
  12. 2025Improving Learning of New Diseases through Knowledge-Enhanced Initialization for Federated Adapter Tuningcited by 0
  13. 2025On the Evolution of Federated Post-Training Large Language Models: A Model Accessibility Viewcited by 0
  14. 2026CoRe-Fed: Bridging Collaborative and Representation Fairness via Federated Embedding Distillationcited by 0
  15. 2026Federated Learning at the Forefront of Fairness: A Multifaceted Perspectivecited by 0
  16. 2026ERIS: Enhancing Privacy and Scalability in Federated Learning via Federated Shard Aggregationcited by 0
  17. 2026Beyond Weights and Gradients: A Taxonomy of Federated Learning Messagescited by 0