mlsanityData to 14 Oct 2026

Convergence Analysis of Aggregation-Broadcast in LoRA-enabled Distributed Fine-Tuning

arXiv 2508.01348 ↗First posted 2 Aug 2025cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) enables collaborative model training across decentralized data sources while preserving data privacy. However, the growing size of Machine Learning (ML) models poses communication and computation challenges in FL. Low-Rank Adaptation (LoRA) has recently been introduced into FL as an efficient fine-tuning method, reducing communication overhead by updating only a small number of trainable parameters. Despite its effectiveness, how to aggregate LoRA-updated local models on the server remains a critical and understudied problem. In this paper, we provide a unified convergence analysis for LoRA-based FL. We first categories the current aggregation method into two major type: Sum-Product (SP) and Product-Sum (PS). Then we formally define the Aggregation-Broadcast Operator (ABO) and derive both weak and strong convergence condition under mild assumptions. Furthermore, we present both weak and strong convergence condition that guarantee convergence of the local model and the global model respectively. These theoretical analyze offer a principled understanding of various aggregation strategies. Notably, we prove that the SP and PS aggregation methods satisfy the weak and strong convergence condition respectively, but differ in their ability to achieve the optimal convergence rate. Extensive experiments on standard benchmarks validate our theoretical findings.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 29 of its 35 references that are in the map. First posted on arXiv 2 Aug 2025.

Builds on

29 of 35 references.

References · 29

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  3. 2019Advances and Open Problems in Federated Learningcited by 3,046
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  5. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  6. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  7. 2018Federated Learning with Non-IID Datacited by 1,108
  8. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  9. 2021Model-Contrastive Federated Learningcited by 766
  10. 2017Federated Multi-Task Learningcited by 672
  11. 2019Fair Resource Allocation in Federated Learningcited by 394
  12. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
17 more
  1. 2020Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategiescited by 198
  2. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
  3. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  4. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
  5. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
  6. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 121
  7. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
  8. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  9. 2019Edge Intelligence: The Confluence of Edge Computing and Artificial Intelligencecited by 76
  10. 2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 38
  11. 2024AutoLoRA: Automatically Tuning Matrix Ranks in Low-Rank Adaptation Based on Meta Learningcited by 31
  12. 2024Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clientscited by 22
  13. 2024FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Modelscited by 18
  14. 2024Towards Efficient Communication and Secure Federated Recommendation System via Low-rank Trainingcited by 14
  15. 2024Robust Federated Finetuning of Foundation Models via Alternating Minimization of LoRAcited by 11
  16. 2025Federated Low-Rank Adaptation for Foundation Models: A Surveycited by 9
  17. 2024RBLA: Rank-Based-LoRA-Aggregation for Fine-tuning Heterogeneous Models in FLaaScited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Subspace-Constrained Federated Learning with Low-Rank Adaptationcited by 0
  2. 2026FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantizationcited by 0