mlsanityData to 14 Oct 2026

Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA

arXiv 2502.01755 ↗First posted 3 Feb 2025cs.LG cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA) optimize federated training by reducing computational and communication costs. We propose RoLoRA, a federated framework using alternating optimization to fine-tune LoRA adapters. Our approach emphasizes the importance of learning up and down projection matrices to enhance expressiveness and robustness. We use both theoretical analysis and extensive experiments to demonstrate the advantages of RoLoRA over prior approaches that either generate imperfect model updates or limit expressiveness of the model. We provide a theoretical analysis on a linear model to highlight the importance of learning both the down-projection and up-projection matrices in LoRA. We validate the insights on a non-linear model and separately provide a convergence proof under general conditions. To bridge theory and practice, we conducted extensive experimental evaluations on language models including RoBERTa-Large, Llama-2-7B on diverse tasks and FL settings to demonstrate the advantages of RoLoRA over other methods.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 19 of its 28 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

19 of 28 references.

References · 19

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  8. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  9. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  10. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  11. 2020Few-Shot Learning via Learning the Representation, Provablycited by 214
  12. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
7 more
  1. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
  2. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 108
  3. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  4. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 86
  5. 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 56
  6. 2021Sample Efficient Linear Meta-Learning by Alternating Minimizationcited by 25
  7. 2024Robust Federated Finetuning of Foundation Models via Alternating Minimization of LoRAcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 13.

17 papers.

Sort:

By area:

  1. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 17
  2. 2025AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projectionscited by 9
  3. 2026FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRAcited by 8
  4. 2025ADF-LoRA: Alternating Low-Rank Aggregation for Decentralized Federated Fine-Tuningcited by 4
  5. 2026FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuningcited by 3
  6. 2026Making Models Unmergeable via Scaling-Sensitive Loss Landscapecited by 1
  7. 2026Preventing Rank Collapse in Federated Low-Rank Adaptation with Client Heterogeneitycited by 1
  8. 2026Stabilized Fine-Tuning with LoRA in Federated Learning: Mitigating the Side Effect of Client Size and Rank via the Scaling Factorcited by 1
  9. 2026Adaptive Selection of LoRA Components in Privacy-Preserving Federated Learningcited by 1
  10. 2026SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Rankscited by 1
  11. 2026FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRAcited by 1
  12. 2025FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoEcited by 0
5 more
  1. 2026Stabilizing Decentralized Federated Fine-Tuning via Topology-Aware Alternating LoRAcited by 0
  2. 2026Collaborative and Efficient Fine-tuning: Leveraging Task Similaritycited by 0
  3. 2026AR1-ZO: Topology-Aware Rank-1 Zeroth-Order Queries for High-Rank LoRA Fine-Tuningcited by 0
  4. 2026Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approachcited by 0
  5. 2026CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregationcited by 0