# Stabilizing Decentralized Federated Fine-Tuning via Topology-Aware Alternating LoRA

arXiv: https://arxiv.org/abs/2602.00451 · HTML: https://mlsanity.com/paper/2602.00451/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 26 references that are in the map.
- First posted on arXiv 31 Jan 2026.

## Builds on (most cited first, 22 of 22)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent](https://mlsanity.com/paper/1705.09056/) (2017)
- [Robust and Communication-Efficient Federated Learning from Non-IID Data](https://mlsanity.com/paper/1903.02891/) (2019)
- [Asynchronous Federated Optimization](https://mlsanity.com/paper/1903.03934/) (2019)
- [Sparse Communication for Distributed Gradient Descent](https://mlsanity.com/paper/1704.05021/) (2017)
- [Improving LoRA in Privacy-preserving Federated Learning](https://mlsanity.com/paper/2403.12313/) (2024)
- [FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations](https://mlsanity.com/paper/2409.05976/) (2024)
- [SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models](https://mlsanity.com/paper/2308.06522/) (2023)
- [Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources](https://mlsanity.com/paper/2402.11505/) (2024)
- [Peer-to-peer Federated Learning on Graphs](https://mlsanity.com/paper/1901.11173/) (2019)
- [Selective Aggregation for Low-Rank Adaptation in Federated Learning](https://mlsanity.com/paper/2410.01463/) (2024)
- [pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuning](https://mlsanity.com/paper/2310.13283/) (2023)
- [Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA](https://mlsanity.com/paper/2502.01755/) (2025)
- [AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections](https://mlsanity.com/paper/2505.12455/) (2025)
- [Decentralized Federated Learning with Model Caching on Mobile Agents](https://mlsanity.com/paper/2408.14001/) (2024)

## Abstract (arXiv)

Decentralized federated learning (DFL), a serverless variant of federated learning, poses unique challenges for parameter-efficient fine-tuning due to the factorized structure of low-rank adaptation (LoRA). Unlike linear parameters, decentralized aggregation of LoRA updates introduces topology-dependent cross terms that can destabilize training under dynamic communication graphs. We propose textttTAD-LoRA, a Topology-Aware Decentralized Low-Rank Adaptation framework that coordinates the updates and mixing of LoRA factors to control inter-client misalignment. We theoretically prove the convergence of textttTAD-LoRA under non-convex objectives, explicitly characterizing the trade-off between topology-induced cross-term error and block-coordinate representation bias governed by the switching interval of alternative training. Experiments under various communication conditions validate our analysis, showing that textttTAD-LoRA achieves robust performance across different communication scenarios, remaining competitive in strongly connected topologies and delivering clear gains under moderately and weakly connected topologies, with particularly strong results on the MNLI dataset.