mlsanityData to 14 Oct 2026

Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach

arXiv 2608.09742 ↗First posted 10 Aug 2026cs.LG cs.AI cs.DCcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Low-rank adaptation (LoRA) represents large language model (LLM) updates with two compact matrix factors, i.e., A and B, providing an efficient way to fine-tune large models in federated learning paradigm. Inspired by the asymmetric roles of the LoRA factors, we study whether A should be shared across clients while B remains client-specific (Share-A/Local-B), or whether B should instead be shared while A remains client-specific (Share-B/Local-A). With a least-squares surrogate, we reveal that Share-A/Local-B requires the client-specific LoRA update matrices to use a common rank-r input-side space, whereas Share-B/Local-A requires a common rank-r output-side space. The two strategies therefore incur different projection residuals, indicating that the preferred strategy is the one with the smaller aggregate residual across clients. With this insight, we propose Federated Adaptive Factor Sharing Low-Rank Adaptation (FedAS-LoRA), which selects the sharing side before training to enhance fine-tuning performance. To enable adaptive factor selection before training, we design a Rank-Aware Shared-Subspace Sufficiency (RSS) metric, which effectively assesses whether a shared rank-r input subspace is sufficient for the local data distributions using representations extracted from a frozen LLM backbone. Experiments across different tasks, data distributions, LoRA ranks, and participation settings confirm the effectiveness of RSS and the superior performance of FedAS-LoRA.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 38 of its 52 references that are in the map. First posted on arXiv 10 Aug 2026.

Builds on

38 of 52 references.

References · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  11. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  12. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
26 more
  1. 2020Adaptive Federated Optimizationcited by 1,046
  2. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  3. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  4. 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
  5. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
  6. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
  7. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  8. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 149
  9. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
  10. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 137
  11. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 134
  12. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 123
  13. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
  14. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  15. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 86
  16. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  17. 2023Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimizationcited by 65
  18. 2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 55
  19. 2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 38
  20. 2023FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Modelscited by 26
  21. 2024Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clientscited by 22
  22. 2025Robust Federated Finetuning of LLMs via Alternating Optimization of LoRAcited by 17
  23. 2024Towards Federated Low-Rank Adaptation of Language Models with Rank Heterogeneitycited by 10
  24. 2025Federated Low-Rank Adaptation for Foundation Models: A Surveycited by 9
  25. 2026FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRAcited by 8
  26. 2024Communication-Efficient Federated Low-Rank Update Algorithm and its Connection to Implicit Regularizationcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation
  2. 2026Federated LoRA Fine-Tuning for LLMs via Collaborative Alignment
  3. 2025Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning
  4. 2026FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA
  5. 2025Communication-Efficient and Accurate Approach for Aggregation in Federated Low-Rank Adaptation
  6. 2024FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Models
  7. 2026FraQ: Efficient Coordinate-Space Recompression for Federated Low-Rank Adaptation
  8. 2026FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment