Flexible Communication for Optimal Distributed Learning over Unpredictable Networks
Not on the map. Why?
Abstract (arXiv)
Gradient compression alleviates expensive communication in distributed deep learning by sending fewer values and its corresponding indices, typically via Allgather (AG). Training with high compression ratio (CR) achieves high accuracy like DenseSGD, but has lower parallel scaling due to high communication cost (i.e., parallel efficiency). Using lower CRs improves parallel efficiency by lowering synchronization cost, but degrades model accuracy as well (statistical efficiency). Further, speedup attained with different models and CRs also varies with network latency, effective bandwidth and collective op used for aggregation. In many cases, collectives like Allreduce (AR) have lower cost than AG to exchange the same amount of data. In this paper, we propose an AR-compatible Topk compressor that is bandwidth-optimal and thus performs better than AG in certain network configurations. We develop a flexible communication strategy that switches between AG and AR based on which collective is optimal in the current settings, and model the pareto-relationship between parallel and statistical efficiency as a multi-objective optimization (MOO) problem to dynamically adjust CR and accelerate training while still converging to high accuracy.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 25 of its 39 references that are in the map. First posted on arXiv 5 Dec 2023.
Builds on
25 of 39 references.
References · 25
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
- 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
- 2018An Empirical Model of Large-Batch Trainingcited by 268
- 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 253
- 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 234
- 2018The Convergence of Sparsified Gradient Methodscited by 228
- 2020pymoo: Multi-objective Optimization in Pythoncited by 196
- 2020The Early Phase of Neural Network Trainingcited by 114
- 2020PyTorch Distributed: Experiences on Accelerating Data Parallel Trainingcited by 87
13 more
- 2017Critical Learning Periods in Deep Neural Networkscited by 78
- 2020Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learningcited by 68
- 2021An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systemscited by 32
- 2021On the Utility of Gradient Compression in Distributed Training Systemscited by 26
- 2020Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusterscited by 23
- 2020Accordion: Adaptive Gradient Communication via Critical Learning Regime Identificationcited by 19
- 2018Large batch size training of neural networks with adversarial training and second-order informationcited by 18
- 2020AdaScale SGD: A User-Friendly Algorithm for Distributed Trainingcited by 16
- 2019Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutescited by 13
- 2023GraVAC: Adaptive Compression for Communication-Efficient Distributed DL Trainingcited by 10
- 2023ScaDLES: Scalable Deep Learning over Streaming data at the Edgecited by 8
- 2023Accelerating Distributed ML Training via Selective Synchronizationcited by 7
- 2023Scavenger: A Cloud Service for Optimizing Cost and Performance of ML Trainingcited by 5
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.