mlsanityData to 7 Oct 2026

Federated Large Language Models: Current Progress and Future Directions

arXiv 2409.15723 ↗First posted 24 Sep 2024cs.LG cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

Large Language Models have achieved impressive performance across diverse applications, yet their training typically depends on centralized data collection, raising serious privacy and governance concerns. Federated Learning offers a decentralized alternative by enabling multiple clients to collaboratively train shared models without exposing raw local data. However, integrating FL with LLMs introduces new challenges, including data heterogeneity, convergence instability, communication overhead, and computational constraints. This survey provides a comprehensive and up-to-date overview of Federated Learning for Large Language Models (FedLLM). We systematically review recent advances, with particular emphasis on federated fine-tuning and federated prompt learning, and analyze how existing methods address efficiency, personalization, and security challenges. We further summarize emerging directions such as federated pre-training and federated agents. Our goal is to offer a structured perspective on this rapidly evolving field and to highlight promising avenues for future research.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 74 of its 111 references that are in the map. First posted on arXiv 24 Sep 2024.

Builds on

74 of 111 references.

References · 74

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  4. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  5. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
  6. 2023Simple synthetic data reduces sycophancy in large language modelscited by 174
  7. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 158
  8. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
  9. 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 141
  10. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 133
  11. 2020How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligencecited by 128
  12. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
38 more
  1. 2021DENSE: Data-Free One-Shot Federated Learningcited by 100
  2. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitationscited by 96
  3. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 92
  4. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
  5. 2022FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learningcited by 86
  6. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  7. 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 83
  8. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 74
  9. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 73
  10. 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
  11. 2022Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learningcited by 71
  12. 2023FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learningcited by 70
  13. 2022Recovering Private Text in Federated Learning of Language Modelscited by 67
  14. 2023Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimizationcited by 65
  15. 2023FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learningcited by 59
  16. 2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
  17. 2023Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generationcited by 54
  18. 2020Federated Learning via Synthetic Datacited by 53
  19. 2022Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Modelscited by 52
  20. 2023Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Modelscited by 49
  21. 2022A Survey on Heterogeneous Federated Learningcited by 42
  22. 2023Prompt Federated Learning for Weather Forecasting: Toward Foundation Models on Meteorological Datacited by 40
  23. 2023FedML-HE: An Efficient Homomorphic-Encryption-Based Privacy-Preserving Federated Learning Systemcited by 38
  24. 2022FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformerscited by 33
  25. 2023Can Public Large Language Models Help Private Cross-device Federated Learning?cited by 30
  26. 2024The Future of Large Language Model Pre-training is Federatedcited by 30
  27. 2022Federated Adaptive Prompt Tuning for Multi-Domain Collaborative Learningcited by 29
  28. 2023Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learningcited by 29
  29. 2024A Survey on Efficient Federated Learning Methods for Foundation Model Trainingcited by 29
  30. 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 26
  31. 2023FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Modelscited by 25
  32. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  33. 2023Low-Parameter Federated Learning with Large Language Modelscited by 22
  34. 2023FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUscited by 20
  35. 2023FedSecurity: Benchmarking Attacks and Defenses in Federated Learning and Federated LLMscited by 19
  36. 2024FeB4RAG: Evaluating Federated Search in the Context of Retrieval Augmented Generationcited by 19
  37. 2024On the Convergence of Zeroth-Order Federated Tuning for Large Language Modelscited by 18
  38. 2023Efficient and Secure Federated Learning for Financial Applicationscited by 17

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 10; 2026: 10.

22 papers.

Sort:

By area:

  1. 2025H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinkingcited by 94
  2. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 65
  3. 2024Federated Learning in Practice: Reflections and Projectionscited by 15
  4. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 15
  5. 2026Skill-R1: Agent Skill Evolution via Reinforcement Learningcited by 12
  6. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  7. 2025Cross-Cloud Data Privacy Protection: Optimizing Collaborative Mechanisms of AI Systems by Integrating Federated Learning and LLMscited by 9
  8. 2025Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuningcited by 9
  9. 2025Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMscited by 2
  10. 2025FLEx: Personalized Federated Learning for Mixture-of-Experts LLMs via Expert Graftingcited by 2
  11. 2026AlienLM: Alienization of Language for API-Boundary Privacy in Black-Box LLMscited by 2
  12. 2024Revisiting Federated Fine-Tuning: A Single Communication Round is Enough for Foundation Modelscited by 1
10 more
  1. 2025A Hybrid Swarm Intelligence Approach for Optimizing Multimodal Large Language Models Deployment in Edge-Cloud-based Federated Learning Environmentscited by 1
  2. 2025Intelligent Orchestration of Distributed Large Foundation Model Inference at the Edgecited by 1
  3. 2026Toward Federated Large Language Models in Medicine: A Parameter-Efficient Framework for Privacy-Preserving, Multi-Institutional Adaptationcited by 1
  4. 2026Empowering Contrastive Federated Sequential Recommendation with LLMscited by 0
  5. 2026GuidaPA: Privacy-Preserving Chatbot for Public Administration via Federated Learningcited by 0
  6. 2026Improving Federated Graph Recommendation with Semantic Guidancecited by 0
  7. 2026TIGER: Inverting Transformer Gradients via Embedding-Subspace Distance Optimizationcited by 0
  8. 2026PFAdapter: Hierarchical LoRA Decomposition for Personalized Federated MLLMscited by 0
  9. 2026FedLNS: Leverage LayerNorm Signature Modeling to Mitigate Adversarial Manipulation in Federated LLMscited by 0
  10. 2026FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantizationcited by 0