# Dithered backprop: A sparse and quantized backpropagation algorithm for more efficient deep neural network training

arXiv: https://arxiv.org/abs/2004.04729 · HTML: https://mlsanity.com/paper/2004.04729/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Apr 2020.

## Built on by (most cited first, 11 of 11)

- [Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formats](https://mlsanity.com/paper/2112.10769/) (2021)
- [No frame left behind: Full Video Action Recognition](https://mlsanity.com/paper/2103.15395/) (2021)
- [L2ight: Enabling On-Chip Learning for Optical Neural Networks via Efficient in-situ Subspace Optimization](https://mlsanity.com/paper/2110.14807/) (2021)
- [Sparse is Enough in Fine-tuning Pre-trained Large Language Models](https://mlsanity.com/paper/2312.11875/) (2023)
- [SparseProp: Efficient Sparse Backpropagation for Faster Training of Neural Networks](https://mlsanity.com/paper/2302.04852/) (2023)
- [Meta-Learning with a Geometry-Adaptive Preconditioner](https://mlsanity.com/paper/2304.01552/) (2023)
- [Zero-Shot Temporal Resolution Domain Adaptation for Spiking Neural Networks](https://mlsanity.com/paper/2411.04760/) (2024)
- [AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks](https://mlsanity.com/paper/2211.03741/) (2022)
- [Less Memory Means smaller GPUs: Backpropagation with Compressed Activations](https://mlsanity.com/paper/2409.11902/) (2024)
- [Unbiased Approximate Vector-Jacobian Products for Efficient Backpropagation](https://mlsanity.com/paper/2602.14701/) (2026)
- [Federated Learning of Spiking Neural Networks under Heterogeneous Temporal Resolutions](https://mlsanity.com/paper/2605.15355/) (2026)

## Abstract (arXiv)

Deep Neural Networks are successful but highly computationally expensive learning systems. One of the main sources of time and energy drains is the well known backpropagation (backprop) algorithm, which roughly accounts for 2/3 of the computational complexity of training. In this work we propose a method for reducing the computational cost of backprop, which we named dithered backprop. It consists in applying a stochastic quantization scheme to intermediate results of the method. The particular quantisation scheme, called non-subtractive dither (NSD), induces sparsity which can be exploited by computing efficient sparse matrix multiplications. Experiments on popular image classification tasks show that it induces 92% sparsity on average across a wide set of models at no or negligible accuracy drop in comparison to state-of-the-art approaches, thus significantly reducing the computational complexity of the backward pass. Moreover, we show that our method is fully compatible to state-of-the-art training methods that reduce the bit-precision of training down to 8-bits, as such being able to further reduce the computational requirements. Finally we discuss and show potential benefits of applying dithered backprop in a distributed training setting, where both communication as well as compute efficiency may increase simultaneously with the number of participant nodes.