mlsanityData to 14 Oct 2026

FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Models

arXiv 2410.09432 ↗First posted 12 Oct 2024cs.DC cs.CL cs.CVcited by 18

Not on the map. Why?

Abstract (arXiv)

Low-Rank Adaptation (LoRA) is a popular technique for efficient fine-tuning of foundation models. However, applying LoRA in federated learning environments, where data is distributed across multiple clients, presents unique challenges. Existing methods rely on traditional federated averaging of LoRA adapters, resulting in inexact updates. To address this, we propose Federated Exact LoRA, or FedEx-LoRA, which adds a residual error term to the pretrained frozen weight matrix. Our approach achieves exact updates with minimal computational and communication overhead, preserving LoRA's efficiency. We evaluate the method on various models across arithmetic reasoning, commonsense reasoning, natural language understanding and natural language generation tasks, showing consistent performance gains over state-of-the-art methods across multiple settings. Through extensive analysis, we quantify that the deviations in updates from the ideal solution are significant, highlighting the need for exact aggregation. Our method's simplicity, efficiency, and broad applicability position it as a promising solution for accurate and effective federated fine-tuning of foundation models. Our code is publicly available at https://github.com/RaghavSinghal10/fedex-lora.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 49 of its 61 references that are in the map. First posted on arXiv 12 Oct 2024.

Builds on

49 of 61 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  10. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
37 more
  1. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  7. 2019Advances and Open Problems in Federated Learningcited by 3,046
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  11. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  12. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  13. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  14. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  15. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  16. 2018Federated Learning with Non-IID Datacited by 1,108
  17. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  18. 2018Neural Network Acceptability Judgmentscited by 958
  19. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  20. 2019Towards Federated Learning at Scale: System Designcited by 832
  21. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  22. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
  23. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  24. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 300
  25. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 253
  26. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  27. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 204
  28. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 200
  29. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
  30. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  31. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 149
  32. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 137
  33. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 136
  34. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 123
  35. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
  36. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 93
  37. 2024PEFT for Speech: Unveiling Optimal Placement, Merging Strategies, and Ensemble Techniquescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 9.

18 papers.

Sort:

By area:

  1. 2025HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 31
  2. 2024Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuningcited by 22
  3. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 17
  4. 2025Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuningcited by 10
  5. 2025Federated Low-Rank Adaptation for Foundation Models: A Surveycited by 9
  6. 2026FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRAcited by 8
  7. 2026FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignmentcited by 5
  8. 2025ABBA-Adapters: Efficient and Expressive Fine-Tuning of Foundation Modelscited by 3
  9. 2025Fed-SE: Federated Self-Evolution for Privacy-Constrained Multi-Environment LLM Agentscited by 3
  10. 2025Convergence Analysis of Aggregation-Broadcast in LoRA-enabled Distributed Fine-Tuningcited by 2
  11. 2025FedLoDrop: Federated LoRA with Dropout for Generalized LLM Fine-tuningcited by 2
  12. 2026Adaptive Selection of LoRA Components in Privacy-Preserving Federated Learningcited by 1
6 more
  1. 2026SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Rankscited by 1
  2. 2026FedPref: Federated Preference Learning for Structured Radiology Report Extractioncited by 1
  3. 2026RW-LoRA: Communication-Efficient Decentralized LoRA Fine-Tuning via Random Walkscited by 1
  4. 2026Collaborative and Efficient Fine-tuning: Leveraging Task Similaritycited by 0
  5. 2026UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Modelscited by 0
  6. 2026Federated LoRA Adaptation of BiomedCLIP Across Four International Chest X-Ray Cohortscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation
  2. 2025Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning
  3. 2026FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA
  4. 2026Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach
  5. 2026FraQ: Efficient Coordinate-Space Recompression for Federated Low-Rank Adaptation
  6. 2025FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoE
  7. 2026Federated LoRA Fine-Tuning for LLMs via Collaborative Alignment
  8. 2025Decentralized Low-Rank Fine-Tuning of Large Language Models