mlsanityData to 7 Oct 2026

ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed Training

arXiv 2104.11125 ↗First posted 21 Apr 2021cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

Large-scale distributed training of Deep Neural Networks (DNNs) on state-of-the-art platforms is expected to be severely communication constrained. To overcome this limitation, numerous gradient compression techniques have been proposed and have demonstrated high compression ratios. However, most existing methods do not scale well to large scale distributed systems (due to gradient build-up) and/or fail to evaluate model fidelity (test accuracy) on large datasets. To mitigate these issues, we propose a new compression technique, Scalable Sparsified Gradient Compression (ScaleCom), that leverages similarity in the gradient distribution amongst learners to provide significantly improved scalability. Using theoretical analysis, we show that ScaleCom provides favorable convergence guarantees and is compatible with gradient all-reduce techniques. Furthermore, we experimentally demonstrate that ScaleCom has small overheads, directly reduces gradient traffic and provides high compression rates (65-400X) and excellent scalability (up to 64 learners and 8-12X larger batch sizes over standard training) across a wide range of applications (image, language, and speech) without significant accuracy loss.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 33 of its 53 references that are in the map. First posted on arXiv 21 Apr 2021.

Builds on

33 of 53 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  9. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  10. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  11. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 496
  12. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
21 more
  1. 2018Sparsified SGD with Memorycited by 373
  2. 2015Deep Learning with Limited Numerical Precisioncited by 368
  3. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 306
  4. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 234
  5. 2018The Convergence of Sparsified Gradient Methodscited by 228
  6. 2018Scaling Neural Machine Translationcited by 204
  7. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 163
  8. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  9. 2019DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compressioncited by 98
  10. 2019Communication-efficient distributed SGD with Sketchingcited by 81
  11. 2018Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimizationcited by 76
  12. 2018Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutescited by 63
  13. 2019Understanding Top-k Sparsification in Distributed Deep Learningcited by 56
  14. 2019A Distributed Synchronous SGD Algorithm with Global Top-k Sparsification for Low Bandwidth Networkscited by 36
  15. 2017AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Trainingcited by 33
  16. 2018GradiVeQ: Vector Quantization for Bandwidth-Efficient Gradient Aggregation in Distributed CNN Trainingcited by 21
  17. 2019The Deep Learning Revolution and Its Implications for Computer Architecture and Chip Designcited by 15
  18. 2017Embedding-Based Speaker Adaptive Training of Deep Neural Networkscited by 7
  19. 2019Realizing Petabyte Scale Acoustic Modelingcited by 6
  20. 2019A Highly Efficient Distributed Deep Learning System For Automatic Speech Recognitioncited by 6
  21. 2020Improving Efficiency in Large-Scale Decentralized Distributed Trainingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 6; 2024: 6; 2025: 3; 2026: 1.

22 papers.

Sort:

  1. 2022Federated Learning with Sparsified Model Perturbation: Improving Accuracy under Client-Level Differential Privacycited by 32
  2. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 29
  3. 2023Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compressioncited by 14
  4. 2024Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real Systemcited by 14
  5. 2022Distributed Adversarial Training to Robustify Deep Neural Networks at Scalecited by 9
  6. 2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Surveycited by 9
  7. 2021Over-the-Air Federated Multi-Task Learning Over MIMO Multiple Access Channelscited by 7
  8. 2024Revisiting Gradient Pruning: A Dual Realization for Defending against Gradient Attackscited by 7
  9. 2022Enabling All In-Edge Deep Learning: A Literature Reviewcited by 4
  10. 2023DropCompute: simple and more robust distributed synchronous training via compute variance reductioncited by 4
  11. 2023Improved Convergence Analysis and SNR Control Strategies for Federated Learning in the Presence of Noisecited by 4
  12. 2023FedREP: A Byzantine-Robust, Communication-Efficient and Privacy-Preserving Framework for Federated Learningcited by 3
10 more
  1. 2024Problem-dependent convergence bounds for randomized linear gradient compressioncited by 3
  2. 2024Novel Gradient Sparsification Algorithm via Bayesian Inferencecited by 2
  3. 2025Compressed Decentralized Momentum Stochastic Gradient Methods for Nonconvex Optimizationcited by 2
  4. 2025EDGC: Entropy-driven Dynamic Gradient Compression for Efficient LLM Trainingcited by 2
  5. 2022Empirical Analysis on Top-k Gradient Sparsification for Distributed Deep Learning in a Supercomputing Environmentcited by 1
  6. 2026TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Trainingcited by 1
  7. 2023DEFT: Exploiting Gradient Norm Difference between Model Layers for Scalable Gradient Sparsificationcited by 0
  8. 2023MiCRO: Near-Zero Cost Gradient Sparsification for Scaling and Accelerating Distributed DNN Trainingcited by 0
  9. 2024Preserving Near-Optimal Gradient Sparsification Cost for Scalable Distributed Deep Learningcited by 0
  10. 2025Regularized Top-k: A Bayesian Framework for Gradient Sparsificationcited by 0