# An In-depth Study of Stochastic Backpropagation

arXiv: https://arxiv.org/abs/2210.00129 · HTML: https://mlsanity.com/paper/2210.00129/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 30 Sep 2022.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [CoAtNet: Marrying Convolution and Attention for All Data Sizes](https://mlsanity.com/paper/2106.04803/) (2021)
- [G-TAD: Sub-Graph Localization for Temporal Action Detection](https://mlsanity.com/paper/1911.11462/) (2019)
- [Sparse Networks from Scratch: Faster Training without Losing Performance](https://mlsanity.com/paper/1907.04840/) (2019)
- [Long Short-Term Transformer for Online Action Detection](https://mlsanity.com/paper/2107.03377/) (2021)
- [Temporal Recurrent Networks for Online Action Detection](https://mlsanity.com/paper/1811.07391/) (2018)
- [A Multigrid Method for Efficiently Training Video Models](https://mlsanity.com/paper/1912.00998/) (2019)
- [Stochastic Backpropagation: A Memory Efficient Strategy for Training Video Models](https://mlsanity.com/paper/2203.16755/) (2022)
- [Sideways: Depth-Parallel Training of Video Models](https://mlsanity.com/paper/2001.06232/) (2020)
- [Gradient Forward-Propagation for Large-Scale Temporal Video Modelling](https://mlsanity.com/paper/2106.08318/) (2021)

## Built on by (most cited first, 2 of 2)

- [Quantum Neural Networks for Power Flow Analysis](https://mlsanity.com/paper/2311.06293/) (2023)
- [SUS backprop: linear backpropagation algorithm for long inputs in transformers](https://mlsanity.com/paper/2505.15080/) (2025)

## Abstract (arXiv)

In this paper, we provide an in-depth study of Stochastic Backpropagation (SBP) when training deep neural networks for standard image classification and object detection tasks. During backward propagation, SBP calculates the gradients by only using a subset of feature maps to save the GPU memory and computational cost. We interpret SBP as an efficient way to implement stochastic gradient decent by performing backpropagation dropout, which leads to considerable memory saving and training process speedup, with a minimal impact on the overall model accuracy. We offer some good practices to apply SBP in training image recognition models, which can be adopted in learning a wide range of deep neural networks. Experiments on image classification and object detection show that SBP can save up to 40% of GPU memory with less than 1% accuracy degradation.