Recent Advances in Federated Learning Driven Large Language Models: A Survey on Architecture, Performance, and Security
On the map: LLM safety and security → Federated unlearning and poisoning
Abstract (arXiv)
Federated Learning (FL) offers a promising paradigm for training Large Language Models (LLMs) in a decentralized manner while preserving data privacy and minimizing communication overhead. This survey examines recent advancements in FL-driven LLMs, with a particular emphasis on architectural designs, performance optimization, and security concerns, including the emerging area of machine unlearning. In this context, machine unlearning refers to the systematic removal of specific data contributions from trained models to comply with privacy regulations such as the Right to be Forgotten. We review a range of strategies enabling unlearning in federated LLMs, including perturbation-based methods, model decomposition, and incremental retraining, while evaluating their trade-offs in terms of efficiency, privacy guarantees, and model utility. Through selected case studies and empirical evaluations, we analyze how these methods perform in practical FL scenarios. This survey identifies critical research directions toward developing secure, adaptable, and high-performing federated LLM systems for real-world deployment.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Federated unlearning and poisoning, one of 12 directions in LLM safety and security. Builds on 41 of its 71 references that are in the map. First posted on arXiv 14 Jun 2024.
Builds on
41 of 71 references.
References · 41
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2023A Survey on Evaluation of Large Language Modelscited by 1,227
- 2021Transformer in Transformercited by 632
- 2021Towards Personalized Federated Learningcited by 462
- 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
- 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
- 2023Role-Play with Large Language Modelscited by 290
- 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 160
- 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
- 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 141
- 2022Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challengescited by 137
- 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 135
29 more
- 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
- 2022Federated Learning with Partial Model Personalizationcited by 114
- 2023SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Researchcited by 112
- 2022Vertical Federated Learning: Concepts, Advances and Challengescited by 104
- 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
- 2021Federated Reconstruction: Partially Local Federated Learningcited by 78
- 2023Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Servicescited by 76
- 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 74
- 2021On Large-Cohort Training for Federated Learningcited by 73
- 2022Recovering Private Text in Federated Learning of Language Modelscited by 67
- 2023Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimizationcited by 65
- 2023FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Modelscited by 62
- 2023Large Language Models Empowered Autonomous Edge AI for Connected Intelligencecited by 55
- 2023ChatGPT and a New Academic Reality: Artificial Intelligence-Written Research Papers and the Ethics of the Large Language Models in Scholarly Publishingcited by 51
- 2024Automatic Speech Recognition using Advanced Deep Learning Approaches: A surveycited by 50
- 2021Scaling Federated Learning for Fine-tuning of Large Language Modelscited by 36
- 2024Mechanics of Next Token Prediction with Self-Attentioncited by 34
- 2019Federated Learning of N-gram Language Modelscited by 33
- 2024An Aggregation-Free Federated Learning for Tackling Data Heterogeneitycited by 33
- 2022Pretrained Models for Multilingual Federated Learningcited by 30
- 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 26
- 2022Scaling Language Model Size in Cross-Device Federated Learningcited by 25
- 2023A Comprehensive Overview and Comparative Analysis on Deep Learning Models: CNN, RNN, LSTM, GRUcited by 25
- 2023Low-Parameter Federated Learning with Large Language Modelscited by 22
- 2024On the Convergence of Zeroth-Order Federated Tuning for Large Language Modelscited by 18
- 2023A Survey for Federated Learning Evaluations: Goals and Measurescited by 16
- 2023Rank-without-GPT: Building GPT-Independent Listwise Rerankers on Open-Source Large Language Modelscited by 14
- 2024Analysis of Privacy Leakage in Federated Large Language Modelscited by 9
- 2024Efficient Language Model Architectures for Differentially Private Federated Learningcited by 1
Built on by
Citing papers per year: 2025: 1.
1 paper.