# Integrated Model, Batch and Domain Parallelism in Training Neural Networks

arXiv: https://arxiv.org/abs/1712.04432 · HTML: https://mlsanity.com/paper/1712.04432/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Dec 2017.

## Built on by (most cited first, 16 of 16)

- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [ResIST: Layer-Wise Decomposition of ResNets for Distributed Training](https://mlsanity.com/paper/2107.00961/) (2021)
- [Sequential Aggregation and Rematerialization: Distributed Full-batch Training of Graph Neural Networks on Large Graphs](https://mlsanity.com/paper/2111.06483/) (2021)
- [THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compression](https://mlsanity.com/paper/2302.08545/) (2023)
- [GIST: Distributed Training for Large-Scale Graph Convolutional Networks](https://mlsanity.com/paper/2102.10424/) (2021)
- [DrJAX: Scalable and Differentiable MapReduce Primitives in JAX](https://mlsanity.com/paper/2403.07128/) (2024)
- [Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training](https://mlsanity.com/paper/2411.13055/) (2024)
- [Model-Parallel Model Selection for Deep Learning Systems](https://mlsanity.com/paper/2107.06469/) (2021)
- [OLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural Networks](https://mlsanity.com/paper/2210.12924/) (2022)
- [Distributed Matrix-Based Sampling for Graph Neural Network Training](https://mlsanity.com/paper/2311.02909/) (2023)
- [Local Critic Training for Model-Parallel Learning of Deep Neural Networks](https://mlsanity.com/paper/2102.01963/) (2021)
- [LOFT: Finding Lottery Tickets through Filter-wise Training](https://mlsanity.com/paper/2210.16169/) (2022)
- [An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networks](https://mlsanity.com/paper/2104.09075/) (2021)
- [Neural Network Methods for Radiation Detectors and Imaging](https://mlsanity.com/paper/2311.05726/) (2023)
- [Scalable Artificial Intelligence for Science: Perspectives, Methods and Exemplars](https://mlsanity.com/paper/2406.17812/) (2024)
- [Inductive Predictions of Extreme Hydrologic Events in The Wabash River Watershed](https://mlsanity.com/paper/2104.14658/) (2021)

## Abstract (arXiv)

We propose a new integrated method of exploiting model, batch and domain parallelism for the training of deep neural networks (DNNs) on large distributed-memory computers using minibatch stochastic gradient descent (SGD). Our goal is to find an efficient parallelization strategy for a fixed batch size using P processes. Our method is inspired by the communication-avoiding algorithms in numerical linear algebra. We see P processes as logically divided into a Pr × Pc grid where the Pr dimension is implicitly responsible for model/domain parallelism and the Pc dimension is implicitly responsible for batch parallelism. In practice, the integrated matrix-based parallel algorithm encapsulates these types of parallelism automatically. We analyze the communication complexity and analytically demonstrate that the lowest communication costs are often achieved neither with pure model nor with pure data parallelism. We also show how the domain parallel approach can help in extending the theoretical scaling limit of the typical batch parallel method.