# A Gradient-Interleaved Scheduler for Energy-Efficient Backpropagation for Training Neural Networks

arXiv: https://arxiv.org/abs/2002.05529 · HTML: https://mlsanity.com/paper/2002.05529/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Feb 2020.

## Built on by (most cited first, 2 of 2)

- [LayerPipe: Accelerating Deep Neural Network Training by Intra-Layer and Inter-Layer Gradient Pipelining and Multiprocessor Scheduling](https://mlsanity.com/paper/2108.06629/) (2021)
- [LayerPipe2: Multistage Pipelining and Weight Recompute via Improved Exponential Moving Average for Training Neural Networks](https://mlsanity.com/paper/2512.08160/) (2025)

## Abstract (arXiv)

This paper addresses design of accelerators using systolic architectures for training of neural networks using a novel gradient interleaving approach. Training the neural network involves backpropagation of error and computation of gradients with respect to the activation functions and weights. It is shown that the gradient with respect to the activation function can be computed using a weight-stationary systolic array while the gradient with respect to the weights can be computed using an output-stationary systolic array. The novelty of the proposed approach lies in interleaving the computations of these two gradients to the same configurable systolic array. This results in reuse of the variables from one computation to the other and eliminates unnecessary memory accesses. The proposed approach leads to 1.4 - 2.2 times savings in terms of number of cycles and 1.9 × savings in terms of memory accesses. Thus, the proposed accelerator reduces latency and energy consumption.