mlsanityData to 7 Oct 2026

FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRA

arXiv 2602.23638 ↗First posted 27 Feb 2026cs.LG cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Federated LoRA provides a communication-efficient mechanism for fine-tuning large language models on decentralized data. In practice, however, a discrepancy between the factor-wise averaging used to preserve low rank and the mathematically correct aggregation of local updates can cause significant aggregation error and unstable training. We argue that a major source of this problem is rotational misalignment, arising from the rotational invariance of low-rank factorizations -- semantically equivalent updates can be represented in different latent subspaces across clients since (Bi Ri)(Ri^top Ai) = Bi Ai. When such misaligned factors are averaged directly, they interfere destructively and degrade the global update. To address this issue, we propose FedRot-LoRA, a federated LoRA framework that aligns client updates via orthogonal transformations prior to aggregation. This alignment preserves the semantic update while reducing cross-client subspace mismatch, without increasing communication cost or restricting model expressivity. We provide a convergence analysis that examines the aggregation error induced by factor-wise averaging and shows how rotational alignment yields a tighter upper bound on this error. Extensive experiments on natural language understanding and generative tasks demonstrate that FedRot-LoRA consistently outperforms existing federated LoRA baselines across a range of heterogeneity levels and LoRA ranks.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 27 of its 36 references that are in the map. First posted on arXiv 27 Feb 2026.

Builds on

27 of 36 references.

References · 27

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  11. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  12. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
15 more
  1. 2020The Future of Digital Health with Federated Learningcited by 625
  2. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  3. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  4. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
  5. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  6. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 158
  7. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
  8. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 144
  9. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  10. 2022FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learningcited by 86
  11. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  12. 2024FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Modelscited by 18
  13. 2025Robust Federated Finetuning of LLMs via Alternating Optimization of LoRAcited by 17
  14. 2024Task-Aware Low-Rank Adaptation of Segment Anything Modelcited by 6
  15. 2025FedSRD: Sparsify-Reconstruct-Decompose for Communication-Efficient Federated Large Language Models Fine-Tuningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Rankscited by 1
  2. 2026FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRAcited by 1
  3. 2026Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learningcited by 0
  4. 2026Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approachcited by 0
  5. 2026Unveiling the Depth-Performance Dilemma in Split-Federated Fine-tuning of LLMscited by 0
  6. 2026RW-LoRA: Communication-Efficient Decentralized LoRA Fine-Tuning via Random Walkscited by 0
  7. 2026What Should Federated LoRA Share? FedSAIL via Input-aware Subspace Alignmentcited by 0
  8. 2026CF-LoRA: Decoupled Factor Aggregation and Adaptation-Aware Client Clustering for Federated LoRA Fine-Tuningcited by 0