Federated Low-Rank Adaptation for Foundation Models: A Survey

arXiv 2505.13502 ↗First posted 16 May 2025cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Effectively leveraging private datasets remains a significant challenge in developing foundation models. Federated Learning (FL) has recently emerged as a collaborative framework that enables multiple users to fine-tune these models while mitigating data privacy risks. Meanwhile, Low-Rank Adaptation (LoRA) offers a resource-efficient alternative for fine-tuning foundation models by dramatically reducing the number of trainable parameters. This survey examines how LoRA has been integrated into federated fine-tuning for foundation models, an area we term FedLoRA, by focusing on three key challenges: distributed learning, heterogeneity, and efficiency. We further categorize existing work based on the specific methods used to address each challenge. Finally, we discuss open research questions and highlight promising directions for future investigation, outlining the next steps for advancing FedLoRA.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 49 of its 60 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

49 of 60 references.

References · 49

2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
2024Large Language Models: A Surveycited by 525
2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 149
2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146

37 more

2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
2023Bayesian Low-rank Adaptation for Large Language Modelscited by 115
2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 93
2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 84
2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 75
2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 74
2024SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 62
2024FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuningcited by 59
2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 55
2023Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Modelscited by 50
2024Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 49
2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 38
2024A Survey on Efficient Federated Learning Methods for Foundation Model Trainingcited by 29
2024FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Expertscited by 27
2023FwdLLM: Efficient FedLLM using Forward Gradientcited by 26
2024FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decompositioncited by 26
2024Federated LoRA with Sparse Communicationcited by 26
2024Federated Fine-Tuning for Pre-Trained Foundation Models Over Wireless Networkscited by 25
2024Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clientscited by 22
2023FedRA: A Random Allocation Strategy for Federated Tuning to Unleash the Power of Heterogeneous Clientscited by 19
2024FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Modelscited by 18
2024Towards Efficient Communication and Secure Federated Recommendation System via Low-rank Trainingcited by 14
2024Federated Adaptation for Foundation Model-based Recommendationscited by 14
2024Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptationcited by 13
2024Robust Federated Finetuning of Foundation Models via Alternating Minimization of LoRAcited by 11
2024Towards Federated Low-Rank Adaptation of Language Models with Rank Heterogeneitycited by 10
2024RBLA: Rank-Based-LoRA-Aggregation for Fine-tuning Heterogeneous Models in FLaaScited by 8
2022Empowering parameter-efficient transfer learning by recognizing the kernel structure in self-attentioncited by 6
2024Communication-Efficient Federated Low-Rank Update Algorithm and its Connection to Implicit Regularizationcited by 6
2024Fisher Information-based Efficient Curriculum Federated Learning with Large Language Modelscited by 6
2024FL-TAC: Enhanced Fine-Tuning in Federated Learning via Low-Rank, Task-Specific Adapter Clusteringcited by 5
2024CG-FedLLM: How to Compress Gradients in Federated Fune-tuning for Large Language Modelscited by 4
2024Personalized Federated Instruction Tuning via Neural Architecture Searchcited by 3
2024FedsLLM: Federated Split Learning for Large Language Models over Communication Networkscited by 3
2024Fed-pilot: Optimizing LoRA Allocation for Efficient Federated Fine-Tuning with Heterogeneous Clientscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.