mlsanityData to 7 Oct 2026

Stabilizing Decentralized Federated Fine-Tuning via Topology-Aware Alternating LoRA

arXiv 2602.00451 ↗First posted 31 Jan 2026cs.LG cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Decentralized federated learning (DFL), a serverless variant of federated learning, poses unique challenges for parameter-efficient fine-tuning due to the factorized structure of low-rank adaptation (LoRA). Unlike linear parameters, decentralized aggregation of LoRA updates introduces topology-dependent cross terms that can destabilize training under dynamic communication graphs. We propose textttTAD-LoRA, a Topology-Aware Decentralized Low-Rank Adaptation framework that coordinates the updates and mixing of LoRA factors to control inter-client misalignment. We theoretically prove the convergence of textttTAD-LoRA under non-convex objectives, explicitly characterizing the trade-off between topology-induced cross-term error and block-coordinate representation bias governed by the switching interval of alternative training. Experiments under various communication conditions validate our analysis, showing that textttTAD-LoRA achieves robust performance across different communication scenarios, remaining competitive in strongly connected topologies and delivering clear gains under moderately and weakly connected topologies, with particularly strong results on the MNLI dataset.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 26 references that are in the map. First posted on arXiv 31 Jan 2026.

Builds on

22 of 26 references.

References · 22

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2019Advances and Open Problems in Federated Learningcited by 3,021
  6. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  7. 2018Federated Learning with Non-IID Datacited by 1,099
  8. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  9. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 496
  10. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 415
  11. 2019Asynchronous Federated Optimizationcited by 264
  12. 2017Sparse Communication for Distributed Gradient Descentcited by 243
10 more
  1. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 158
  2. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 144
  3. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  4. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 92
  5. 2019Peer-to-peer Federated Learning on Graphscited by 85
  6. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  7. 2023pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuningcited by 38
  8. 2025Robust Federated Finetuning of LLMs via Alternating Optimization of LoRAcited by 17
  9. 2025AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projectionscited by 8
  10. 2024Decentralized Federated Learning with Model Caching on Mobile Agentscited by 4

Built on by

Not cited yet.