mlsanityData to 14 Oct 2026

MergeComp: A Compression Scheduler for Scalable Communication-Efficient Distributed Training

arXiv 2103.15195 ↗First posted 28 Mar 2021cs.DC cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Large-scale distributed training is increasingly becoming communication bound. Many gradient compression algorithms have been proposed to reduce the communication overhead and improve scalability. However, it has been observed that in some cases gradient compression may even harm the performance of distributed training. In this paper, we propose MergeComp, a compression scheduler to optimize the scalability of communication-efficient distributed training. It automatically schedules the compression operations to optimize the performance of compression algorithms without the knowledge of model architectures or system parameters. We have applied MergeComp to nine popular compression algorithms. Our evaluations show that MergeComp can improve the performance of compression algorithms by up to 3.83x without losing accuracy. It can even achieve a scaling factor of distributed training up to 99% over high-speed networks.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 33 of its 56 references that are in the map. First posted on arXiv 28 Mar 2021.

Builds on

33 of 56 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  5. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  6. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  7. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 502
  8. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
  9. 2018Sparsified SGD with Memorycited by 376
  10. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 310
  11. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 279
  12. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 256
21 more
  1. 2017Sparse Communication for Distributed Gradient Descentcited by 249
  2. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
  3. 2018The Convergence of Sparsified Gradient Methodscited by 229
  4. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 202
  5. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 201
  6. 2015MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systemscited by 195
  7. 2019Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computationscited by 181
  8. 2020Is Local SGD Better than Minibatch SGD?cited by 151
  9. 2016QSGD: Communication-Efficient SGD via Gradient Quantization and Encodingcited by 147
  10. 2018Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimizationcited by 76
  11. 2019Scaling Distributed Machine Learning with In-Network Aggregationcited by 65
  12. 2017Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusterscited by 57
  13. 20158-Bit Approximations for Parallelism in Deep Learningcited by 56
  14. 2019Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedbackcited by 50
  15. 2019Blink: Fast and Generic Collectives for Distributed MLcited by 36
  16. 2018SparCML: High-Performance Sparse Communication for Machine Learningcited by 34
  17. 2017AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Trainingcited by 33
  18. 2018Variance-based Gradient Compression for Efficient Distributed Deep Learningcited by 22
  19. 2020Is Network the Bottleneck of Distributed Training?cited by 20
  20. 2018Parameter Hub: a Rack-Scale Parameter Server for Distributed Deep Neural Network Trainingcited by 17
  21. 20183LC: Lightweight and Effective Traffic Compression for Distributed Machine Learningcited by 16

Built on by

Cited by year

Citing papers per year: 2022: 1.

1 paper.

  1. 2022ByteComp: Revisiting Gradient Compression in Distributed Trainingcited by 1