# LayerPipe: Accelerating Deep Neural Network Training by Intra-Layer and Inter-Layer Gradient Pipelining and Multiprocessor Scheduling

arXiv: https://arxiv.org/abs/2108.06629 · HTML: https://mlsanity.com/paper/2108.06629/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 14 Aug 2021.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)
- [Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis](https://mlsanity.com/paper/1802.09941/) (2018)
- [Decoupled Neural Interfaces using Synthetic Gradients](https://mlsanity.com/paper/1608.05343/) (2016)
- [Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools](https://mlsanity.com/paper/1903.11314/) (2019)
- [Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks](https://mlsanity.com/paper/1705.01626/) (2017)
- [Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training](https://mlsanity.com/paper/1811.03619/) (2018)
- [Training Neural Networks Using Features Replay](https://mlsanity.com/paper/1807.04511/) (2018)
- [SparseNN: An Energy-Efficient Neural Network Accelerator Exploiting Input and Output Sparsity](https://mlsanity.com/paper/1711.01263/) (2017)
- [A Gradient-Interleaved Scheduler for Energy-Efficient Backpropagation for Training Neural Networks](https://mlsanity.com/paper/2002.05529/) (2020)
- [BaPipe: Exploration of Balanced Pipeline Parallelism for DNN Training](https://mlsanity.com/paper/2012.12544/) (2020)

## Built on by (most cited first, 2 of 2)

- [A Vision Toward Energy-Efficient Domain-Specific Artificial Intelligence Models and Agents](https://mlsanity.com/paper/2510.22052/) (2025)
- [LayerPipe2: Multistage Pipelining and Weight Recompute via Improved Exponential Moving Average for Training Neural Networks](https://mlsanity.com/paper/2512.08160/) (2025)

## Abstract (arXiv)

The time required for training the neural networks increases with size, complexity, and depth. Training model parameters by backpropagation inherently creates feedback loops. These loops hinder efficient pipelining and scheduling of the tasks within the layer and between consecutive layers. Prior approaches, such as PipeDream, have exploited the use of delayed gradient to achieve inter-layer pipelining. However, these approaches treat the entire backpropagation as a single task; this leads to an increase in computation time and processor underutilization. This paper presents novel optimization approaches where the gradient computations with respect to the weights and the activation functions are considered independently; therefore, these can be computed in parallel. This is referred to as intra-layer optimization. Additionally, the gradient computation with respect to the activation function is further divided into two parts and distributed to two consecutive layers. This leads to balanced scheduling where the computation time of each layer is the same. This is referred to as inter-layer optimization. The proposed system, referred to as LayerPipe, reduces the number of clock cycles required for training while maximizing processor utilization with minimal inter-processor communication overhead. LayerPipe achieves an average speedup of 25% and upwards of 80% with 7 to 9 processors with less communication overhead when compared to PipeDream.