mlsanityData to 14 Oct 2026

Federated Dynamical Low-Rank Training with Global Loss Convergence Guarantees

arXiv 2406.17887 ↗First posted 25 Jun 2024cs.LG cs.AI math.OCcited by 7

Not on the map. Why?

Abstract (arXiv)

In this work, we propose a federated dynamical low-rank training (FeDLRT) scheme to reduce client compute and communication costs - two significant performance bottlenecks in horizontal federated learning. Our method builds upon dynamical low-rank splitting schemes for manifold-constrained optimization to create a global low-rank basis of network weights, which enables client training on a small coefficient matrix. A consistent global low-rank basis allows us to incorporate a variance correction scheme and prove global loss descent and convergence to a stationary point. Dynamic augmentation and truncation of the low-rank bases automatically optimizes computing and communication resource utilization. We demonstrate the efficiency of FeDLRT in an array of computer vision benchmarks and show a reduction of client compute and communication costs by up to an order of magnitude with minimal impacts on global accuracy.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 39 of its 51 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

39 of 51 references.

References · 39

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  4. 2019Advances and Open Problems in Federated Learningcited by 3,046
  5. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  6. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  7. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  8. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 564
  9. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  10. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  11. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 356
  12. 2018Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edgecited by 352
27 more
  1. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 306
  2. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 284
  3. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
  4. 2018Federated Learning via Over-the-Air Computationcited by 219
  5. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
  6. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 203
  7. 2020Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategiescited by 198
  8. 2019FedHealth: A Federated Transfer Learning Framework for Wearable Healthcarecited by 182
  9. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 141
  10. 2013Communication Efficient Distributed Optimization using an Approximate Newton-type Methodcited by 132
  11. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 123
  12. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 123
  13. 2020FedSplit: An algorithmic framework for fast federated optimizationcited by 108
  14. 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
  15. 2019Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizerscited by 61
  16. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  17. 2021Initialization and Regularization of Factorized Neural Layerscited by 58
  18. 2022ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsitycited by 45
  19. 2022Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equationscited by 43
  20. 2023Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learningcited by 41
  21. 2020Federated Transfer Learning: concept and applicationscited by 32
  22. 2021FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorizationcited by 31
  23. 2017Compressing Recurrent Neural Network with Tensor Traincited by 28
  24. 2023Understanding Why ViT Trains Badly on Small Datasets: An Intuitive Perspectivecited by 28
  25. 2020FedSKETCH: Communication-Efficient and Private Federated Learning via Sketchingcited by 20
  26. 2021Communication-Efficient Federated Learning with Dual-Side Low-Rank Compressioncited by 10
  27. 2023Riemannian Low-Rank Model Compression for Federated Learning with Over-the-Air Aggregationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 1.

7 papers.

  1. 2025Quantum-PEFT: Ultra parameter-efficient fine-tuningcited by 11
  2. 2025Dynamical Low-Rank Compression of Neural Networks with Robustness under Adversarial Attackscited by 6
  3. 2024GeoLoRA: Geometric integration for parameter efficient fine-tuningcited by 5
  4. 2025A geometric framework for momentum-based optimizers for low-rank trainingcited by 5
  5. 2025Mitigating Subject Dependency in EEG Decoding with Subject-Specific Low-Rank Adapterscited by 4
  6. 2025BLaST: High Performance Inference and Pretraining using BLock Sparse Transformerscited by 3
  7. 2026Compressing Vision Transformers in Geospatial Transfer Learning with Manifold-Constrained Optimizationcited by 0