# Minimal Effort Back Propagation for Convolutional Neural Networks

arXiv: https://arxiv.org/abs/1709.05804 · HTML: https://mlsanity.com/paper/1709.05804/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2017.

## Built on by (most cited first, 7 of 7)

- [Fine-Tuning Language Models with Just Forward Passes](https://mlsanity.com/paper/2305.17333/) (2023)
- [Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models](https://mlsanity.com/paper/2404.08080/) (2024)
- [To Adapt or Not to Adapt? Real-Time Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2307.15063/) (2023)
- [Dynamic Collective Intelligence Learning: Finding Efficient Sparse Model via Refined Gradients for Pruned Weights](https://mlsanity.com/paper/2109.04660/) (2021)
- [TinyProp -- Adaptive Sparse Backpropagation for Efficient TinyML On-device Learning](https://mlsanity.com/paper/2308.09201/) (2023)
- [Advancing On-Device Neural Network Training with TinyPropv2: Dynamic, Sparse, and Efficient Backpropagation](https://mlsanity.com/paper/2409.07109/) (2024)
- [ssProp: Energy-Efficient Training for Convolutional Neural Networks with Scheduled Sparse Back Propagation](https://mlsanity.com/paper/2408.12561/) (2024)

## Abstract (arXiv)

As traditional neural network consumes a significant amount of computing resources during back propagation, citetSun2017mePropSB propose a simple yet effective technique to alleviate this problem. In this technique, only a small subset of the full gradients are computed to update the model parameters. In this paper we extend this technique into the Convolutional Neural Network(CNN) to reduce calculation in back propagation, and the surprising results verify its validity in CNN: only 5% of the gradients are passed back but the model still achieves the same effect as the traditional CNN, or even better. We also show that the top-k selection of gradients leads to a sparse calculation in back propagation, which may bring significant computational benefits for high computational complexity of convolution operation in CNN.