mlsanityData to 14 Oct 2026

The Panaceas for Improving Low-Rank Decomposition in Communication-Efficient Federated Learning

arXiv 2505.23176 ↗First posted 29 May 2025cs.LG cs.DCcited by 5

Not on the map. Why?

Abstract (arXiv)

To improve the training efficiency of federated learning (FL), previous research has employed low-rank decomposition techniques to reduce communication overhead. In this paper, we seek to enhance the performance of these low-rank decomposition methods. Specifically, we focus on three key issues related to decomposition in FL: what to decompose, how to decompose, and how to aggregate. Subsequently, we introduce three novel techniques: Model Update Decomposition (MUD), Block-wise Kronecker Decomposition (BKD), and Aggregation-Aware Decomposition (AAD), each targeting a specific issue. These techniques are complementary and can be applied simultaneously to achieve optimal performance. Additionally, we provide a rigorous theoretical analysis to ensure the convergence of the proposed MUD. Extensive experimental results show that our approach achieves faster convergence and superior accuracy compared to relevant baseline methods. The code is available at https://github.com/Leopold1423/fedmud-icml25.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 40 of its 56 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

40 of 56 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  5. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  6. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  7. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  8. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 387
  9. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  10. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 356
  11. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 310
  12. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
28 more
  1. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  2. 2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 163
  3. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  4. 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 133
  5. 2024A Survey on LoRA of Large Language Modelscited by 67
  6. 2021Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Bettercited by 55
  7. 2022ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsitycited by 45
  8. 2019On Generalization Bounds of a Family of Recurrent Neural Networkscited by 42
  9. 2021DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learningcited by 41
  10. 2020Adaptive Federated Dropout: Improving Communication Efficiency and Generalization for Federated Learningcited by 38
  11. 2022Sparse Random Networks for Communication-Efficient Federated Learningcited by 32
  12. 2021FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorizationcited by 31
  13. 2020On the Global Convergence of Training Deep Linear ResNetscited by 29
  14. 2022EF21-P and Friends: Improved Theoretical Communication Complexity for Distributed Optimization with Bidirectional Compressioncited by 28
  15. 2023DoCoFL: Downlink Compression for Cross-Device Federated Learningcited by 28
  16. 2023D-Separation for Causal Self-Explanationcited by 21
  17. 2023Enhancing the Rationale-Input Alignment for Self-explaining Rationalizationcited by 15
  18. 2024Decoupling General and Personalized Knowledge in Federated Learning via Additive and Low-Rank Decompositioncited by 15
  19. 2021Communication-Efficient Federated Learning with Binary Neural Networkscited by 14
  20. 2024Embedding Compression in Recommender Systems: A Surveycited by 14
  21. 2023MGR: Multi-generator Based Rationalizationcited by 12
  22. 2022Adaptive Low-Precision Training for Embeddings in Click-Through Rate Predictioncited by 11
  23. 2024Is the MMI Criterion Necessary for Interpretability? Degenerating Non-causal Features to Plain Noise for Self-Rationalizationcited by 11
  24. 2024FedBAT: Communication-Efficient Federated Learning via Learnable Binarizationcited by 9
  25. 2022Federated Progressive Sparsification (Purge, Merge, Tune)+cited by 8
  26. 2025Breaking Free from MMI: A New Frontier in Rationalization by Probing Input Utilizationcited by 6
  27. 2024Masked Random Noise for Communication Efficient Federated Learningcited by 4
  28. 2024Mixed-Precision Embeddings for Large-Scale Recommendation Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptationcited by 6
  2. 2025BoRA: Towards More Expressive Low-Rank Adaptation with Block Diversitycited by 4
  3. 2025Quantifying Distributional Invariance in Causal Subgraph for IRM-Free Graph Generalizationcited by 2
  4. 2025FedBiF: Communication-Efficient Federated Learning via Bits Freezingcited by 1
  5. 2026Federated Learning of Binary Neural Networks: Enabling Low-Cost Inferencecited by 0