mlsanityData to 14 Oct 2026

CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation

arXiv 2609.33147 ↗First posted 27 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated low-rank adaptation (LoRA) enables collaborative fine-tuning of large language models without centralizing private client data. Its factorized update, however, creates a structural mismatch in federated averaging: averaging the two LoRA factors separately does not equal averaging their products. Existing exact methods resolve this issue mainly by freezing an entire factor or alternating factors across rounds, but none can update factors simultaneously without aggregation errors or expanding communication ranks. To address this fundamental problem, we present textttCFLoRA, a federated LoRA scheme that partitions latent LoRA channels into two complementary sets in every communication round. By ensuring that columns and rows are complementary across factors, we eliminate bilinear terms in matrix multiplications, making federated aggregation exact. Crucially, our framework also supports clients with heterogeneous rank budgets. Convergence analysis validates textttCFLoRA achieves O(1/√T) convergence rate of the textitoriginal LoRA objective in homogeneous-rank cases. Extensive experiments with RoBERTa on the GLUE benchmark and with LLaMA-3.2-3B-Instruct on commonsense reasoning tasks demonstrate that textttCFLoRA achieves superior performance and training efficiency compared to state-of-the-art federated LoRA baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 33 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

30 of 33 references.

References · 30

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2019Advances and Open Problems in Federated Learningcited by 3,046
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  11. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  12. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
18 more
  1. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  2. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
  3. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  4. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
  5. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
  6. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  7. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  8. 2024Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 49
  9. 2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 38
  10. 2024Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clientscited by 22
  11. 2025Robust Federated Finetuning of LLMs via Alternating Optimization of LoRAcited by 17
  12. 2024Towards Federated Low-Rank Adaptation of Language Models with Rank Heterogeneitycited by 10
  13. 2025Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuningcited by 9
  14. 2026FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignmentcited by 5
  15. 2025Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMscited by 2
  16. 2026Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRAcited by 2
  17. 2026Adaptive Selection of LoRA Components in Privacy-Preserving Federated Learningcited by 1
  18. 2026FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRAcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Federated LoRA Fine-Tuning for LLMs via Collaborative Alignment
  2. 2026FraQ: Efficient Coordinate-Space Recompression for Federated Low-Rank Adaptation
  3. 2025Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning
  4. 2026Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach
  5. 2024FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Models
  6. 2025Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning
  7. 2025Decentralized Low-Rank Fine-Tuning of Large Language Models
  8. 2026FLoKD: Adaptive Knowledge Distillation for Federated Low-Rank LLM over Wireless Networks