# Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks

arXiv: https://arxiv.org/abs/1802.04924 · HTML: https://mlsanity.com/paper/1802.04924/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Feb 2018.

## Built on by (most cited first, 23 of 23)

- [TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models](https://mlsanity.com/paper/2102.07988/) (2021)
- [Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism](https://mlsanity.com/paper/2211.13878/) (2022)
- [On the Utility of Gradient Compression in Distributed Training Systems](https://mlsanity.com/paper/2103.00543/) (2021)
- [OneFlow: Redesign the Distributed Deep Learning Framework from Scratch](https://mlsanity.com/paper/2110.15032/) (2021)
- [DreamShard: Generalizable Embedding Table Placement for Recommender Systems](https://mlsanity.com/paper/2210.02023/) (2022)
- [Improving Automatic Parallel Training via Balanced Memory Workload Optimization](https://mlsanity.com/paper/2307.02031/) (2023)
- [BagPipe: Accelerating Deep Recommendation Model Training](https://mlsanity.com/paper/2202.12429/) (2022)
- [Automatic Graph Partitioning for Very Large-scale Deep Learning](https://mlsanity.com/paper/2103.16063/) (2021)
- [Local Critic Training for Model-Parallel Learning of Deep Neural Networks](https://mlsanity.com/paper/2102.01963/) (2021)
- [A Survey of Large-Scale Deep Learning Serving System Optimization: Challenges and Opportunities](https://mlsanity.com/paper/2111.14247/) (2021)
- [End-to-end Adaptive Distributed Training on PaddlePaddle](https://mlsanity.com/paper/2112.02752/) (2021)
- [Pre-train and Search: Efficient Embedding Table Sharding with Pre-trained Neural Cost Models](https://mlsanity.com/paper/2305.01868/) (2023)
- [Partitioning sparse deep neural networks for scalable training and inference](https://mlsanity.com/paper/2104.11805/) (2021)
- [Enabling Large Batch Size Training for DNN Models Beyond the Memory Limit While Maintaining Performance](https://mlsanity.com/paper/2110.12484/) (2021)
- [An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networks](https://mlsanity.com/paper/2104.09075/) (2021)
- [Collage: Seamless Integration of Deep Learning Backends with Automatic Placement](https://mlsanity.com/paper/2111.00655/) (2021)
- [Layer-Wise Partitioning and Merging for Efficient and Scalable Deep Learning](https://mlsanity.com/paper/2207.11019/) (2022)
- [DISCO: Distributed Inference with Sparse Communications](https://mlsanity.com/paper/2302.11180/) (2023)
- [UniAP: Unifying Inter- and Intra-Layer Automatic Parallelism by Mixed Integer Quadratic Programming](https://mlsanity.com/paper/2307.16375/) (2023)
- [A Survey on Design Methodologies for Accelerating Deep Learning on Heterogeneous Architectures](https://mlsanity.com/paper/2311.17815/) (2023)
- [PaSE: Parallelization Strategies for Efficient DNN Training](https://mlsanity.com/paper/2407.04001/) (2024)
- [Flexible Coded Distributed Convolution Computing for Enhanced Straggler Resilience and Numerical Stability in Distributed CNNs](https://mlsanity.com/paper/2411.01579/) (2024)
- [Automatic Operator-level Parallelism Planning for Distributed Deep Learning -- A Mixed-Integer Programming Approach](https://mlsanity.com/paper/2503.09357/) (2025)

## Abstract (arXiv)

The past few years have witnessed growth in the computational requirements for training deep convolutional neural networks. Current approaches parallelize training onto multiple devices by applying a single parallelization strategy (e.g., data or model parallelism) to all layers in a network. Although easy to reason about, these approaches result in suboptimal runtime performance in large-scale distributed training, since different layers in a network may prefer different parallelization strategies. In this paper, we propose layer-wise parallelism that allows each layer in a network to use an individual parallelization strategy. We jointly optimize how each layer is parallelized by solving a graph search problem. Our evaluation shows that layer-wise parallelism outperforms state-of-the-art approaches by increasing training throughput, reducing communication costs, achieving better scalability to multiple GPUs, while maintaining original network accuracy.