mlsanityData to 14 Oct 2026

ADF-LoRA: Alternating Low-Rank Aggregation for Decentralized Federated Fine-Tuning

arXiv 2511.18291 ↗First posted 23 Nov 2025cs.LG cs.DCcited by 4

Not on the map. Why?

Abstract (arXiv)

This paper revisits alternating low-rank updates for federated fine-tuning and examines their behavior in decentralized federated learning (DFL). While alternating the LoRA matrices has been shown to stabilize aggregation in centralized FL, extending this mechanism to decentralized, peer-to-peer communication introduces new challenges due to phase-state mismatch and block-wise divergence across clients. We introduce ADF-LoRA, which synchronizes the update of only one low-rank matrix per round and mixes both matrices to maintain more consistent parameter states under decentralized propagation. This design preserves the cross-term suppression effect of alternating updates while improving stability in serverless topologies. We provide a convergence analysis under standard smoothness assumptions and evaluate ADF-LoRA on multiple GLUE tasks. Experiments show that ADF-LoRA achieves faster and smoother convergence and delivers the highest average accuracy across tasks, outperforming existing LoRA variants in decentralized FL by a consistent margin.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 23 of its 26 references that are in the map. First posted on arXiv 23 Nov 2025.

Builds on

23 of 26 references.

References · 23

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2019Advances and Open Problems in Federated Learningcited by 3,046
  6. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  7. 2018Federated Learning with Non-IID Datacited by 1,108
  8. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  9. 2019On the Convergence of Adam and Beyondcited by 841
  10. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 502
  11. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  12. 2019Asynchronous Federated Optimizationcited by 267
11 more
  1. 2017Sparse Communication for Distributed Gradient Descentcited by 249
  2. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  3. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
  4. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 93
  5. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  6. 2019Peer-to-peer Federated Learning on Graphscited by 85
  7. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  8. 2023pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuningcited by 40
  9. 2025Robust Federated Finetuning of LLMs via Alternating Optimization of LoRAcited by 17
  10. 2025AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projectionscited by 9
  11. 2024Decentralized Federated Learning with Model Caching on Mobile Agentscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRAcited by 2
  2. 2026SeedFlood: A Step Toward Scalable Decentralized Fine-Tuning of LLMscited by 1
  3. 2026Priority-Aware Learning-Unlearning Correction for Dynamic Decentralized LoRA Fine-Tuningcited by 0
  4. 2026FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantizationcited by 0