mlsanityData to 14 Oct 2026

Communication-Efficient Federated Low-Rank Update Algorithm and its Connection to Implicit Regularization

arXiv 2409.12371 ↗First posted 19 Sep 2024cs.LG cs.AI stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) faces significant challenges related to communication efficiency and performance reduction when scaling to many clients. To address these issues, we explore the potential of using low-rank updates and provide the first theoretical study of rank properties in FL. Our theoretical analysis shows that a client's loss exhibits a higher-rank structure (i.e., gradients span higher-rank subspaces of the Hessian) compared to the server's loss, and that low-rank approximations of the clients' gradients have greater similarity. Based on this insight, we hypothesize that constraining client-side optimization to a low-rank subspace could provide an implicit regularization effect while reducing communication costs. Consequently, we propose FedLoRU, a general low-rank update framework for FL. Our framework enforces low-rank client-side updates and accumulates these updates to form a higher-rank model. We are able to establish convergence of the algorithm; the convergence rate matches FedAvg. Additionally, variants of FedLoRU can adapt to environments with statistical and model heterogeneity by employing multiple or hierarchical low-rank updates. Experimental results demonstrate that FedLoRU performs comparably to full-rank algorithms and exhibits robustness to heterogeneous and large numbers of clients.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 43 of its 66 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

43 of 66 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2019Advances and Open Problems in Federated Learningcited by 3,046
  9. 2020Adaptive Federated Optimizationcited by 1,046
  10. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  11. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  12. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
31 more
  1. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 370
  2. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 356
  3. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  4. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
  5. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  6. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  7. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  8. 2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 163
  9. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 161
  10. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  11. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 159
  12. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 154
  13. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 141
  14. 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 133
  15. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 128
  16. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
  17. 2024Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learningcited by 82
  18. 2021Adaptive Quantization of Model Updates for Communication-Efficient Federated Learningcited by 63
  19. 2020Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Trainingcited by 50
  20. 2020Design and Analysis of Uplink and Downlink Communications for Federated Learningcited by 48
  21. 2021Communication Efficiency in Federated Learning: Achievements and Challengescited by 40
  22. 2021FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorizationcited by 31
  23. 2024Federated LoRA with Sparse Communicationcited by 26
  24. 2024MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuningcited by 19
  25. 2022Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank?cited by 16
  26. 2018Asymmetry Helps: Eigenvalue and Eigenvector Analyses of Asymmetrically Perturbed Low-Rank Matricescited by 10
  27. 2021Communication-Efficient Federated Learning with Dual-Side Low-Rank Compressioncited by 10
  28. 2022Federated Optimization of Smooth Loss Functionscited by 6
  29. 2023Unveiling the Hessian's Connection to the Decision Boundarycited by 5
  30. 2022Universal characteristics of deep neural network loss surfaces from random matrix theorycited by 4
  31. 2021On the Impact of Stable Ranks in Deep Netscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 1.

6 papers.

  1. 2024DeMo: Decoupled Momentum Optimizationcited by 11
  2. 2025On the Convergence Rate of LoRA Gradient Descentcited by 10
  3. 2025Federated Low-Rank Adaptation for Foundation Models: A Surveycited by 9
  4. 2025An Efficient Subspace Algorithm for Federated Learning on Heterogeneous Datacited by 4
  5. 2025Federated Low-Rank Tensor Estimation for Multimodal Image Reconstructioncited by 0
  6. 2026Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approachcited by 0