# Recent Advances in Federated Learning Driven Large Language Models: A Survey on Architecture, Performance, and Security

arXiv: https://arxiv.org/abs/2406.09831 · HTML: https://mlsanity.com/paper/2406.09831/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Federated unlearning and poisoning, one of 12 directions in LLM safety and security.
- Builds on 41 of its 71 references that are in the map.
- First posted on arXiv 14 Jun 2024.

## Placed in

- LLM safety and security / [Federated unlearning and poisoning](https://mlsanity.com/safety/federated-unlearning-and-poisoning/)

## Builds on (most cited first, 30 of 41)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [Towards Personalized Federated Learning](https://mlsanity.com/paper/2103.00710/) (2021)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining](https://mlsanity.com/paper/2305.10429/) (2023)
- [Role-Play with Large Language Models](https://mlsanity.com/paper/2305.16367/) (2023)
- [Heterogeneous Federated Learning: State-of-the-art and Research Challenges](https://mlsanity.com/paper/2307.10616/) (2023)
- [FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning](https://mlsanity.com/paper/2309.00363/) (2023)
- [Federated Learning from Pre-Trained Models: A Contrastive Learning Approach](https://mlsanity.com/paper/2209.10083/) (2022)
- [Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challenges](https://mlsanity.com/paper/2211.08413/) (2022)
- [FedScale: Benchmarking Model and System Performance of Federated Learning at Scale](https://mlsanity.com/paper/2105.11367/) (2021)
- [OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning](https://mlsanity.com/paper/2402.06954/) (2024)
- [SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research](https://mlsanity.com/paper/2308.13149/) (2023)
- [Federated Learning with Partial Model Personalization](https://mlsanity.com/paper/2204.03809/) (2022)
- [Vertical Federated Learning: Concepts, Advances and Challenges](https://mlsanity.com/paper/2211.12814/) (2022)
- [Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmark](https://mlsanity.com/paper/2311.06750/) (2023)
- [Federated Reconstruction: Partially Local Federated Learning](https://mlsanity.com/paper/2102.03448/) (2021)
- [Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services](https://mlsanity.com/paper/2303.16129/) (2023)
- [When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methods](https://mlsanity.com/paper/2212.10025/) (2022)
- [On Large-Cohort Training for Federated Learning](https://mlsanity.com/paper/2106.07820/) (2021)
- [Recovering Private Text in Federated Learning of Language Models](https://mlsanity.com/paper/2205.08514/) (2022)
- [Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization](https://mlsanity.com/paper/2310.15080/) (2023)
- [FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Models](https://mlsanity.com/paper/2310.10049/) (2023)
- [Large Language Models Empowered Autonomous Edge AI for Connected Intelligence](https://mlsanity.com/paper/2307.02779/) (2023)
- [ChatGPT and a New Academic Reality: Artificial Intelligence-Written Research Papers and the Ethics of the Large Language Models in Scholarly Publishing](https://mlsanity.com/paper/2303.13367/) (2023)
- [Automatic Speech Recognition using Advanced Deep Learning Approaches: A survey](https://mlsanity.com/paper/2403.01255/) (2024)
- [Scaling Federated Learning for Fine-tuning of Large Language Models](https://mlsanity.com/paper/2102.00875/) (2021)
- [Mechanics of Next Token Prediction with Self-Attention](https://mlsanity.com/paper/2403.08081/) (2024)
- [Federated Learning of N-gram Language Models](https://mlsanity.com/paper/1910.03432/) (2019)

## Built on by (most cited first, 1 of 1)

- [Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions](https://mlsanity.com/paper/2503.16585/) (2025)

## Abstract (arXiv)

Federated Learning (FL) offers a promising paradigm for training Large Language Models (LLMs) in a decentralized manner while preserving data privacy and minimizing communication overhead. This survey examines recent advancements in FL-driven LLMs, with a particular emphasis on architectural designs, performance optimization, and security concerns, including the emerging area of machine unlearning. In this context, machine unlearning refers to the systematic removal of specific data contributions from trained models to comply with privacy regulations such as the Right to be Forgotten. We review a range of strategies enabling unlearning in federated LLMs, including perturbation-based methods, model decomposition, and incremental retraining, while evaluating their trade-offs in terms of efficiency, privacy guarantees, and model utility. Through selected case studies and empirical evaluations, we analyze how these methods perform in practical FL scenarios. This survey identifies critical research directions toward developing secure, adaptable, and high-performing federated LLM systems for real-world deployment.