Data to 14 Oct 2026

PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization

arXiv 1905.13727 ↗First posted 31 May 2019cs.LG cs.DC math.OCcited by 238

Not on the map. Why?

Abstract (arXiv)

We study gradient compression methods to alleviate the communication bottleneck in data-parallel distributed optimization. Despite the significant attention received, current compression schemes either do not scale well or fail to achieve the target test accuracy. We propose a new low-rank gradient compressor based on power iteration that can i) compress gradients rapidly, ii) efficiently aggregate the compressed gradients using all-reduce, and iii) achieve test performance on par with SGD. The proposed algorithm is the only method evaluated that achieves consistent wall-clock speedups when benchmarked against regular SGD with an optimized communication backend. We demonstrate reduced training times for convolutional networks as well as LSTMs on common datasets. Our code is available at https://github.com/epfml/powersgd.

Key facts. Cited by 238 papers in the map (data through 14 Oct 2026). First posted on arXiv 31 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 40; 2022: 32; 2023: 36; 2024: 39; 2025: 39; 2026: 52.

238 papers.

Sort:

By area:

2021Zero-Shot Text-to-Image Generationcited by 4,288
2021A Field Guide to Federated Optimizationcited by 315
2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 158
2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 141
2021Large Scale Private Learning via Low-rank Reparametrizationcited by 88
2021Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learningcited by 86
2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 79
20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 75
2025Dion: Distributed Orthonormalized Updatescited by 68
2021Provably Secure Federated Learning against Malicious Clientscited by 63
2021Adaptive Quantization of Model Updates for Communication-Efficient Federated Learningcited by 63

38 more

2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 59
2021DRIVE: One-bit Distributed Mean Estimationcited by 52
2021FedNL: Making Newton-Type Methods Applicable to Federated Learningcited by 52
2022Generalizing to New Physical Systems via Context-Informed Dynamics Modelcited by 50
2021Rethinking gradient sparsification as total error minimizationcited by 41
2022A Survey on Gradient Inversion: Attacks, Defenses and Future Directionscited by 40
2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 38
2025MuonBP: Faster Muon via Block-Periodic Orthogonalizationcited by 35
2022Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compressioncited by 33
2021An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systemscited by 32
2021Permutation Compressors for Provably Faster Distributed Nonconvex Optimizationcited by 32
2022Sparse Random Networks for Communication-Efficient Federated Learningcited by 32
2023Momentum Provably Improves Error Feedback!cited by 31
2023DoCoFL: Downlink Compression for Cross-Device Federated Learningcited by 28
2025Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunchcited by 28
2022projUNN: efficient method for training deep networks with unitary matricescited by 27
2025Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCocited by 27
2021On the Utility of Gradient Compression in Distributed Training Systemscited by 26
20211-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speedcited by 24
2021ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed Trainingcited by 22
2023Reducing Communication for Split Learning by Randomized Top-k Sparsificationcited by 22
2021Smoothness Matrices Beat Smoothness Constants: Better Communication Compression Techniques for Distributed Optimizationcited by 21
2022FedShuffle: Recipes for Better Use of Local Work in Federated Learningcited by 21
2021Trade-offs of Local SGD at Scale: An Empirical Studycited by 20
2022Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learningcited by 20
2022Communication Acceleration of Local Gradient Methods via an Accelerated Primal-Dual Algorithm with Inexact Proxcited by 19
2022GradSkip: Communication-Accelerated Local Gradient Methods with Better Computational Complexitycited by 18
2024Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradientscited by 18
2025MuLoCo: Muon is a practical inner optimizer for DiLoCocited by 18
2023z-SignFedAvg: A Unified Stochastic Sign-based Compression for Federated Learningcited by 17
2025Generalized Gradient Norm Clipping & Non-Euclidean (L0,L1)-Smoothnesscited by 17
2022Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank?cited by 16
2021Distributed Methods with Compressed Communication for Solving Variational Inequalities, with Theoretical Guaranteescited by 15
2022FedLite: A Scalable Approach for Federated Learning on Resource-constrained Clientscited by 15
2023Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compressioncited by 15
2023EControl: Fast Distributed Optimization with Compression and Error Controlcited by 15

50 most cited of 238 All 238 (JSON)