mlsanityData to 14 Oct 2026

ByteComp: Revisiting Gradient Compression in Distributed Training

arXiv 2205.14465 ↗First posted 28 May 2022cs.LG cs.DCcited by 1

Not on the map. Why?

Abstract (arXiv)

Gradient compression (GC) is a promising approach to addressing the communication bottleneck in distributed deep learning (DDL). However, it is challenging to find the optimal compression strategy for applying GC to DDL because of the intricate interactions among tensors. To fully unleash the benefits of GC, two questions must be addressed: 1) How to express all compression strategies and the corresponding interactions among tensors of any DDL training job? 2) How to quickly select a near-optimal compression strategy? In this paper, we propose ByteComp to answer these questions. It first designs a decision tree abstraction to express all the compression strategies and develops empirical models to timeline tensor computation, communication, and compression to enable ByteComp to derive the intricate interactions among tensors. It then designs a compression decision algorithm that analyzes tensor interactions to eliminate and prioritize strategies and optimally offloads compression to CPUs. Experimental evaluations show that ByteComp can improve the training throughput over the start-of-the-art compression-enabled system by up to 77% for representative DDL training jobs. Moreover, the computational time needed to select the compression strategy is measured in milliseconds, and the selected strategy is only a few percent from optimal.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 46 of its 76 references that are in the map. First posted on arXiv 28 May 2022.

Builds on

46 of 76 references.

References · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  8. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  9. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  10. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
  11. 2018Sparsified SGD with Memorycited by 376
  12. 2015Skip-Thought Vectorscited by 322
34 more
  1. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 310
  2. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 279
  3. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 256
  4. 2017Sparse Communication for Distributed Gradient Descentcited by 249
  5. 2014cuDNN: Efficient Primitives for Deep Learningcited by 245
  6. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  7. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 201
  8. 2020Ansor: Generating High-Performance Tensor Programs for Deep Learningcited by 201
  9. 2015MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systemscited by 195
  10. 2019U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translationcited by 186
  11. 2016QSGD: Communication-Efficient SGD via Gradient Quantization and Encodingcited by 147
  12. 2017Revisiting Distributed Synchronous SGDcited by 142
  13. 2018Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimizationcited by 76
  14. 2018Glow: Graph Lowering Compiler Techniques for Neural Networkscited by 75
  15. 2019Scaling Distributed Machine Learning with In-Network Aggregationcited by 65
  16. 2018TVM: An Automated End-to-End Optimizing Compiler for Deep Learningcited by 62
  17. 2017Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusterscited by 57
  18. 20158-Bit Approximations for Parallelism in Deep Learningcited by 56
  19. 2019Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedbackcited by 50
  20. 2019Blink: Fast and Generic Collectives for Distributed MLcited by 36
  21. 2018SparCML: High-Performance Sparse Communication for Machine Learningcited by 34
  22. 2017AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Trainingcited by 33
  23. 2018TicTac: Accelerating Distributed Deep Learning with Communication Schedulingcited by 27
  24. 2021On the Utility of Gradient Compression in Distributed Training Systemscited by 26
  25. 2020Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusterscited by 23
  26. 2019Priority-based Parameter Propagation for Distributed DNN Trainingcited by 21
  27. 2020Is Network the Bottleneck of Distributed Training?cited by 20
  28. 2018Parameter Hub: a Rack-Scale Parameter Server for Distributed Deep Neural Network Trainingcited by 17
  29. 2019CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Serverscited by 14
  30. 2019Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutescited by 13
  31. 2016DLAU: A Scalable Deep Learning Accelerator Unit on FPGAcited by 12
  32. 2021Compressed Communication for Distributed Training: Adaptive Methods and Systemcited by 5
  33. 2021MergeComp: A Compression Scheduler for Scalable Communication-Efficient Distributed Trainingcited by 1
  34. 2021MAPA: Multi-Accelerator Pattern Allocation Policy for Multi-Tenant GPU Serverscited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023Evaluation and Optimization of Gradient Compression for Distributed Deep Learningcited by 7