# Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA

arXiv: https://arxiv.org/abs/2502.01755 · HTML: https://mlsanity.com/paper/2502.01755/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 28 references that are in the map.
- First posted on arXiv 3 Feb 2025.

## Builds on (most cited first, 19 of 19)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [Exploiting Shared Representations for Personalized Federated Learning](https://mlsanity.com/paper/2102.07078/) (2021)
- [LoRA+: Efficient Low Rank Adaptation of Large Models](https://mlsanity.com/paper/2402.12354/) (2024)
- [Few-Shot Learning via Learning the Representation, Provably](https://mlsanity.com/paper/2002.09434/) (2020)
- [Improving LoRA in Privacy-preserving Federated Learning](https://mlsanity.com/paper/2403.12313/) (2024)
- [FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations](https://mlsanity.com/paper/2409.05976/) (2024)
- [Flora: Low-Rank Adapters Are Secretly Gradient Compressors](https://mlsanity.com/paper/2402.03293/) (2024)
- [Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources](https://mlsanity.com/paper/2402.11505/) (2024)
- [Asymmetry in Low-Rank Adapters of Foundation Models](https://mlsanity.com/paper/2402.16842/) (2024)
- [FedAvg with Fine Tuning: Local Updates Lead to Representation Learning](https://mlsanity.com/paper/2205.13692/) (2022)
- [Sample Efficient Linear Meta-Learning by Alternating Minimization](https://mlsanity.com/paper/2105.08306/) (2021)
- [Robust Federated Finetuning of Foundation Models via Alternating Minimization of LoRA](https://mlsanity.com/paper/2409.02346/) (2024)

## Built on by (most cited first, 17 of 17)

- [A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learning](https://mlsanity.com/paper/2504.21099/) (2025)
- [AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections](https://mlsanity.com/paper/2505.12455/) (2025)
- [FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRA](https://mlsanity.com/paper/2602.23638/) (2026)
- [ADF-LoRA: Alternating Low-Rank Aggregation for Decentralized Federated Fine-Tuning](https://mlsanity.com/paper/2511.18291/) (2025)
- [FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuning](https://mlsanity.com/paper/2603.08014/) (2026)
- [Making Models Unmergeable via Scaling-Sensitive Loss Landscape](https://mlsanity.com/paper/2601.21898/) (2026)
- [Preventing Rank Collapse in Federated Low-Rank Adaptation with Client Heterogeneity](https://mlsanity.com/paper/2602.13486/) (2026)
- [Stabilized Fine-Tuning with LoRA in Federated Learning: Mitigating the Side Effect of Client Size and Rank via the Scaling Factor](https://mlsanity.com/paper/2603.08058/) (2026)
- [Adaptive Selection of LoRA Components in Privacy-Preserving Federated Learning](https://mlsanity.com/paper/2605.05769/) (2026)
- [SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Ranks](https://mlsanity.com/paper/2608.10144/) (2026)
- [FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA](https://mlsanity.com/paper/2608.15381/) (2026)
- [FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoE](https://mlsanity.com/paper/2506.16600/) (2025)
- [Stabilizing Decentralized Federated Fine-Tuning via Topology-Aware Alternating LoRA](https://mlsanity.com/paper/2602.00451/) (2026)
- [Collaborative and Efficient Fine-tuning: Leveraging Task Similarity](https://mlsanity.com/paper/2602.07218/) (2026)
- [AR1-ZO: Topology-Aware Rank-1 Zeroth-Order Queries for High-Rank LoRA Fine-Tuning](https://mlsanity.com/paper/2605.19767/) (2026)
- [Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach](https://mlsanity.com/paper/2608.09742/) (2026)
- [CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation](https://mlsanity.com/paper/2609.33147/) (2026)

## Abstract (arXiv)

Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA) optimize federated training by reducing computational and communication costs. We propose RoLoRA, a federated framework using alternating optimization to fine-tune LoRA adapters. Our approach emphasizes the importance of learning up and down projection matrices to enhance expressiveness and robustness. We use both theoretical analysis and extensive experiments to demonstrate the advantages of RoLoRA over prior approaches that either generate imperfect model updates or limit expressiveness of the model. We provide a theoretical analysis on a linear model to highlight the importance of learning both the down-projection and up-projection matrices in LoRA. We validate the insights on a non-linear model and separately provide a convergence proof under general conditions. To bridge theory and practice, we conducted extensive experimental evaluations on language models including RoBERTa-Large, Llama-2-7B on diverse tasks and FL settings to demonstrate the advantages of RoLoRA over other methods.