# Accelerated CNN Training Through Gradient Approximation

arXiv: https://arxiv.org/abs/1908.05460 · HTML: https://mlsanity.com/paper/1908.05460/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Aug 2019.

## Built on by (most cited first, 3 of 3)

- [Deep Neural Networks Based Weight Approximation and Computation Reuse for 2-D Image Classification](https://mlsanity.com/paper/2105.02954/) (2021)
- [XConv: Low-memory stochastic backpropagation for convolutional layers](https://mlsanity.com/paper/2106.06998/) (2021)
- [ssProp: Energy-Efficient Training for Convolutional Neural Networks with Scheduled Sparse Back Propagation](https://mlsanity.com/paper/2408.12561/) (2024)

## Abstract (arXiv)

Training deep convolutional neural networks such as VGG and ResNet by gradient descent is an expensive exercise requiring specialized hardware such as GPUs. Recent works have examined the possibility of approximating the gradient computation while maintaining the same convergence properties. While promising, the approximations only work on relatively small datasets such as MNIST. They also fail to achieve real wall-clock speedups due to lack of efficient GPU implementations of the proposed approximation methods. In this work, we explore three alternative methods to approximate gradients, with an efficient GPU kernel implementation for one of them. We achieve wall-clock speedup with ResNet-20 and VGG-19 on the CIFAR-10 dataset upwards of 7%, with a minimal loss in validation accuracy.