# Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform

arXiv: https://arxiv.org/abs/1809.02839 · HTML: https://mlsanity.com/paper/1809.02839/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Sep 2018.

## Built on by (most cited first, 28 of 28)

- [ResIST: Layer-Wise Decomposition of ResNets for Distributed Training](https://mlsanity.com/paper/2107.00961/) (2021)
- [Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression](https://mlsanity.com/paper/2301.09830/) (2023)
- [Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real System](https://mlsanity.com/paper/2403.06664/) (2024)
- [mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs](https://mlsanity.com/paper/2312.02515/) (2023)
- [Amazon SageMaker Model Parallelism: A General and Flexible Framework for Large Model Training](https://mlsanity.com/paper/2111.05972/) (2021)
- [Weight Prediction Boosts the Convergence of AdamW](https://mlsanity.com/paper/2302.00195/) (2023)
- [Model-Distributed Inference for Large Language Models at the Edge](https://mlsanity.com/paper/2505.18164/) (2025)
- [Automatic Graph Partitioning for Very Large-scale Deep Learning](https://mlsanity.com/paper/2103.16063/) (2021)
- [RevBiFPN: The Fully Reversible Bidirectional Feature Pyramid Network](https://mlsanity.com/paper/2206.14098/) (2022)
- [ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training](https://mlsanity.com/paper/2406.02613/) (2024)
- [Nesterov Method for Asynchronous Pipeline Parallel Optimization](https://mlsanity.com/paper/2505.01099/) (2025)
- [XGrad: Boosting Gradient-Based Optimizers With Weight Prediction](https://mlsanity.com/paper/2305.18240/) (2023)
- [Parareal Neural Networks Emulating a Parallel-in-time Algorithm](https://mlsanity.com/paper/2103.08802/) (2021)
- [PipeOptim: Ensuring Effective 1F1B Schedule with Optimizer-Dependent Weight Prediction](https://mlsanity.com/paper/2312.00839/) (2023)
- [LOFT: Finding Lottery Tickets through Filter-wise Training](https://mlsanity.com/paper/2210.16169/) (2022)
- [AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism](https://mlsanity.com/paper/2601.22442/) (2026)
- [DISCO: Distributed Inference with Sparse Communications](https://mlsanity.com/paper/2302.11180/) (2023)
- [Priority-Aware Model-Distributed Inference at Edge Networks](https://mlsanity.com/paper/2412.12371/) (2024)
- [UniCAIM: A Unified CAM/CIM Architecture with Static-Dynamic KV Cache Pruning for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2504.07479/) (2025)
- [Mitigating Staleness in Asynchronous Pipeline Parallelism via Basis Rotation](https://mlsanity.com/paper/2602.03515/) (2026)
- [AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Training](https://mlsanity.com/paper/2605.29664/) (2026)
- [A Survey From Distributed Machine Learning to Distributed Deep Learning](https://mlsanity.com/paper/2307.05232/) (2023)
- [Toward efficient resource utilization at edge nodes in federated learning](https://mlsanity.com/paper/2309.10367/) (2023)
- [Cyclic Data Parallelism for Efficient Parallelism of Deep Neural Networks](https://mlsanity.com/paper/2403.08837/) (2024)
- [Efficient Training on Multiple Consumer GPUs with RoundPipe](https://mlsanity.com/paper/2604.27085/) (2026)
- [Gradient Extrapolation-Based Policy Optimization](https://mlsanity.com/paper/2605.06755/) (2026)
- [Breaking the Bubble: Asynchronous Pipeline Parallel Training with Bounded Weight Inconsistency](https://mlsanity.com/paper/2606.07881/) (2026)
- [One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining](https://mlsanity.com/paper/2606.30634/) (2026)

## Abstract (arXiv)

The training process of Deep Neural Network (DNN) is compute-intensive, often taking days to weeks to train a DNN model. Therefore, parallel execution of DNN training on GPUs is a widely adopted approach to speed up the process nowadays. Due to the implementation simplicity, data parallelism is currently the most commonly used parallelization method. Nonetheless, data parallelism suffers from excessive inter-GPU communication overhead due to frequent weight synchronization among GPUs. Another approach is pipelined model parallelism, which partitions a DNN model among GPUs, and processes multiple mini-batches concurrently. This approach can significantly reduce inter-GPU communication cost compared to data parallelism. However, pipelined model parallelism faces the weight staleness issue; that is, gradients are computed with stale weights, leading to training instability and accuracy loss. In this paper, we present a pipelined model parallel execution method that enables high GPU utilization while maintaining robust training accuracy via a novel weight prediction technique, SpecTrain. Experimental results show that our proposal achieves up to 8.91x speedup compared to data parallelism on a 4-GPU platform while maintaining comparable model accuracy.