FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregation
Not on the map. Why?
Abstract (arXiv)
Federated Learning (FL) facilitates the fine-tuning of Foundation Models (FMs) using distributed data sources, with Low-Rank Adaptation (LoRA) gaining popularity due to its low communication costs and strong performance. While recent work acknowledges the benefits of heterogeneous LoRA in FL and introduces flexible algorithms to support its implementation, our theoretical analysis reveals a critical gap: existing methods lack formal convergence guarantees due to parameter truncation and biased gradient updates. Specifically, adapting client-specific LoRA ranks necessitates truncating global parameters, which introduces inherent truncation errors and leads to subsequent inaccurate gradient updates that accumulate over training rounds, ultimately degrading performance. To address the above issues, we propose FedHL, a simple yet effective Federated Learning framework tailored for Heterogeneous LoRA. By leveraging the full-rank global model as a calibrated aggregation basis, FedHL eliminates the direct truncation bias from initial alignment with client-specific ranks. Furthermore, we derive the theoretically optimal aggregation weights by minimizing the gradient drift term in the convergence upper bound. Our analysis shows that FedHL guarantees O(1/√T) convergence rate, and experiments on multiple real-world datasets demonstrate a 1-3% improvement over several state-of-the-art methods.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 37 of its 50 references that are in the map. First posted on arXiv 24 May 2025.
Builds on
37 of 50 references.
References · 37
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
- 2018Local SGD Converges Fast and Communicates Littlecited by 605
- 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
- 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
- 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
- 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
25 more
- 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
- 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
- 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 149
- 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
- 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 134
- 2022A Kernel-Based View of Language Model Fine-Tuningcited by 121
- 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
- 2023The Expressive Power of Low-Rank Adaptationcited by 102
- 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 84
- 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 74
- 2022Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learningcited by 71
- 2024A Survey on LoRA of Large Language Modelscited by 67
- 2024FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuningcited by 59
- 2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
- 2023Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Modelscited by 49
- 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
- 2023Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learningcited by 41
- 2023pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuningcited by 40
- 2024Federated LoRA with Sparse Communicationcited by 26
- 2024Federated Fine-Tuning for Pre-Trained Foundation Models Over Wireless Networkscited by 25
- 2023Low-Parameter Federated Learning with Large Language Modelscited by 23
- 2025Ten Challenging Problems in Federated Foundation Modelscited by 20
- 2024On the Convergence of Zeroth-Order Federated Tuning for Large Language Modelscited by 19
- 2024AutoRE: Document-Level Relation Extraction with Large Language Modelscited by 16
- 2024Ferret: Federated Full-Parameter Tuning at Scale for Large Language Modelscited by 6
Built on by
Citing papers per year: 2026: 3.
3 papers.