mlsanityData to 7 Oct 2026

FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations

arXiv 2409.05976 ↗First posted 9 Sep 2024cs.LG cs.DCcited by 144

Not on the map. Why?

Abstract (arXiv)

The rapid development of Large Language Models (LLMs) has been pivotal in advancing AI, with pre-trained LLMs being adaptable to diverse downstream tasks through fine-tuning. Federated learning (FL) further enhances fine-tuning in a privacy-aware manner by utilizing clients' local data through in-situ computation, eliminating the need for data movement. However, fine-tuning LLMs, given their massive scale of parameters, poses challenges for clients with constrained and heterogeneous resources in FL. Previous methods employed low-rank adaptation (LoRA) for efficient federated fine-tuning but utilized traditional FL aggregation strategies on LoRA adapters. These approaches led to mathematically inaccurate aggregation noise, reducing fine-tuning effectiveness and failing to address heterogeneous LoRAs. In this work, we first highlight the mathematical incorrectness of LoRA aggregation in existing federated fine-tuning methods. We introduce a new approach called FLORA that enables federated fine-tuning on heterogeneous LoRA adapters across clients through a novel stacking-based aggregation method. Our approach is noise-free and seamlessly supports heterogeneous LoRA adapters. Extensive experiments demonstrate FLORA' s superior performance in both homogeneous and heterogeneous settings, surpassing state-of-the-art methods. We envision this work as a milestone for efficient, privacy-preserving, and accurate federated fine-tuning of LLMs. Our code is available at https://github.com/ATP-1010/FederatedLLM.

Key facts. Cited by 144 papers in the map (data through 7 Oct 2026). Builds on 24 of its 41 references that are in the map. First posted on arXiv 9 Sep 2024.

Builds on

24 of 41 references.

References · 24

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  10. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  11. 2018Federated Learning with Non-IID Datacited by 1,099
  12. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
12 more
  1. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  2. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  3. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  4. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
  5. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
  6. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 114
  7. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  8. 2024A Survey on LoRA of Large Language Modelscited by 67
  9. 2024SHED: Shapley-Based Automated Dataset Refinement for Instruction Fine-Tuningcited by 19
  10. 2023MerA: Merging Pretrained Adapters For Few-Shot Learningcited by 13
  11. 2023FedHyper: A Universal and Robust Learning Rate Scheduler for Federated Learning with Hypergradient Descentcited by 8
  12. 2024COALA: A Practical and Vision-Centric Federated Learning Platformcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 71; 2026: 63.

144 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  3. 2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 37
  4. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  5. 2025HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 30
  6. 2024Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clientscited by 22
  7. 2025Robust Federated Finetuning of LLMs via Alternating Optimization of LoRAcited by 17
  8. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
  9. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 15
  10. 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRAcited by 13
  11. 2024Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Modelscited by 12
  12. 2025Decentralized Low-Rank Fine-Tuning of Large Language Modelscited by 12
38 more
  1. 2025FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRAcited by 11
  2. 2025FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Modelscited by 11
  3. 2026HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Expertscited by 11
  4. 2025Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuningcited by 10
  5. 2025Federated Continual Instruction Tuningcited by 10
  6. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  7. 2025EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devicescited by 10
  8. 2025Federated Low-Rank Adaptation for Foundation Models: A Surveycited by 9
  9. 2025Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuningcited by 9
  10. 2025AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projectionscited by 8
  11. 2026FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRAcited by 8
  12. 2025FSL-SAGE: Accelerating Federated Split Learning via Smashed Activation Gradient Estimationcited by 7
  13. 2025Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocationcited by 7
  14. 2025LLMs meet Federated Learning for Scalable and Secure IoT Managementcited by 6
  15. 2025Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptationcited by 5
  16. 2025FedMentor: Domain-Aware Differential Privacy for Heterogeneous Federated LLMs in Mental Healthcited by 5
  17. 2025Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuningcited by 5
  18. 2025FedSRD: Sparsify-Reconstruct-Decompose for Communication-Efficient Federated Large Language Models Fine-Tuningcited by 5
  19. 2025A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthinesscited by 5
  20. 2026FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignmentcited by 5
  21. 2025Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Modelscited by 4
  22. 2025A Shared Low-Rank Adaptation Approach to Personalized RLHFcited by 4
  23. 2025FLoRIST: Singular Value Thresholding for Efficient and Accurate Federated Fine-Tuning of Large Language Modelscited by 4
  24. 2025Learning Like Humans: Resource-Efficient Federated Fine-Tuning through Cognitive Developmental Stagescited by 4
  25. 2025DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Modelscited by 4
  26. 2025Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuningcited by 4
  27. 2025ADF-LoRA: Alternating Low-Rank Aggregation for Decentralized Federated Fine-Tuningcited by 4
  28. 2026Rethinking LoRA for Privacy-Preserving Federated Learning in Large Modelscited by 4
  29. 2024Fed-pilot: Optimizing LoRA Allocation for Efficient Federated Fine-Tuning with Heterogeneous Clientscited by 3
  30. 2025ABBA-Adapters: Efficient and Expressive Fine-Tuning of Foundation Modelscited by 3
  31. 2025FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregationcited by 3
  32. 2026Federate the Router: Learning Language Model Routers with Sparse and Decentralized Evaluationscited by 3
  33. 2026FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuningcited by 3
  34. 2024HeteroTune: Efficient Federated Learning for Large Heterogeneous Modelscited by 2
  35. 2025Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMscited by 2
  36. 2025Adaptive Weighted Parameter Fusion with CLIP for Class-Incremental Learningcited by 2
  37. 2025Communication-Efficient and Personalized Federated Foundation Model Fine-Tuning via Tri-Matrix Adaptationcited by 2
  38. 2025Two Stage Wireless Federated LoRA Fine-Tuning with Sparsified Orthogonal Updatescited by 2

50 most cited of 144 All 144 (JSON)