mlsanityData to 14 Oct 2026

From PowerSGD to PowerSGD+: Low-Rank Gradient Compression for Distributed Optimization with Convergence Guarantees

arXiv 2509.11254 ↗First posted 14 Sep 2025math.OC cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Low-rank gradient compression methods, such as PowerSGD, have gained attention in communication-efficient distributed optimization. However, the convergence guarantees of PowerSGD remain unclear, particularly in stochastic settings. In this paper, we show that PowerSGD does not always converge to the optimal solution and provide a clear counterexample to support this finding. To address this, we introduce PowerSGD+, which periodically updates the projection subspace via singular value decomposition, ensuring that it remains aligned with the optimal subspace. We prove that PowerSGD+ converges under standard assumptions and validate its effectiveness through empirical evaluation on large language model tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 25 references that are in the map. First posted on arXiv 14 Sep 2025.

Builds on

18 of 25 references.

References · 18

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  3. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  4. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  5. 2018Sparsified SGD with Memorycited by 376
  6. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 310
  7. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 256
  8. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
  9. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 201
  10. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 136
  11. 2021EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedbackcited by 133
  12. 2024Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structurescited by 50
6 more
  1. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 45
  2. 2024Subspace Optimization for Large Language Models with Convergence Guaranteescited by 34
  3. 2022Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compressioncited by 33
  4. 2023Momentum Provably Improves Error Feedback!cited by 31
  5. 2025Greedy Low-Rank Gradient Compression for Distributed Learning with Convergence Guaranteescited by 11
  6. 2023Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencentcited by 8

Built on by

Not cited yet.