An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networks
Not on the map. Why?
Abstract (arXiv)
Deep Neural Network (DNN) frameworks use distributed training to enable faster time to convergence and alleviate memory capacity limitations when training large models and/or using high dimension inputs. With the steady increase in datasets and model sizes, model/hybrid parallelism is deemed to have an important role in the future of distributed training of DNNs. We analyze the compute, communication, and memory requirements of Convolutional Neural Networks (CNNs) to understand the trade-offs between different parallelism approaches on performance and scalability. We leverage our model-driven analysis to be the basis for an oracle utility which can help in detecting the limitations and bottlenecks of different parallelism approaches at scale. We evaluate the oracle on six parallelization strategies, with four CNN models and multiple datasets (2D and 3D), on up to 1024 GPUs. The results demonstrate that the oracle has an average accuracy of about 86.74% when compared to empirical results, and as high as 97.57% for data parallelism.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 31 of its 55 references that are in the map. First posted on arXiv 19 Apr 2021.
Builds on
31 of 55 references.
References · 31
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
- 2018Group Normalizationcited by 1,514
- 2016Training Deep Nets with Sublinear Memory Costcited by 825
- 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
- 2018Context Encoding for Semantic Segmentationcited by 289
- 2014One weird trick for parallelizing convolutional neural networkscited by 264
- 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
19 more
- 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 154
- 2018Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutescited by 63
- 2019PruneTrain: Fast Neural Network Training by Dynamic Sparse Model Reconfigurationcited by 41
- 2019Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirectcited by 41
- 2015AtomNet: A Deep Convolutional Neural Network for Bioactivity Prediction in Structure-based Drug Discoverycited by 38
- 2018SparCML: High-Performance Sparse Communication for Machine Learningcited by 34
- 2020Automatic Cross-Replica Sharding of Weight Update in Data-Parallel Trainingcited by 30
- 2018Exascale Deep Learning for Climate Analyticscited by 28
- 2020torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Modelscited by 26
- 2018Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networkscited by 23
- 2020Convolutional Neural Network Training with Distributed K-FACcited by 22
- 2017Integrated Model, Batch and Domain Parallelism in Training Neural Networkscited by 16
- 2018CosmoFlow: Using Deep Learning to Learn the Universe at Scalecited by 14
- 2019Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 secondscited by 12
- 2017ChainerMN: Scalable Distributed Deep Learning Frameworkcited by 10
- 2019Improving Strong-Scaling of CNN Training by Exploiting Finer-Grained Parallelismcited by 8
- 2020Scaling Distributed Deep Learning Workloads beyond the Memory Capacity with KARMAcited by 5
- 2019Benanza: Automatic μBenchmark Generation to Compute "Lower-bound" Latency and Inform Optimizations of Deep Learning Models on GPUscited by 3
- 2020Extracting Clean Performance Models from Tainted Programscited by 2
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.