# TinyProp -- Adaptive Sparse Backpropagation for Efficient TinyML On-device Learning

arXiv: https://arxiv.org/abs/2308.09201 · HTML: https://mlsanity.com/paper/2308.09201/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 26 references that are in the map.
- First posted on arXiv 17 Aug 2023.

## Builds on (most cited first, 12 of 12)

- [TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning](https://mlsanity.com/paper/1705.07878/) (2017)
- [Direct Feedback Alignment Provides Learning in Deep Neural Networks](https://mlsanity.com/paper/1609.01596/) (2016)
- [On Using Very Large Target Vocabulary for Neural Machine Translation](https://mlsanity.com/paper/1412.2007/) (2014)
- [TinyOL: TinyML with Online-Learning on Microcontrollers](https://mlsanity.com/paper/2103.08295/) (2021)
- [Biologically-plausible learning algorithms can scale to large datasets](https://mlsanity.com/paper/1811.03567/) (2018)
- [meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting](https://mlsanity.com/paper/1706.06197/) (2017)
- [TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning](https://mlsanity.com/paper/2007.11622/) (2020)
- [Faster Neural Network Training with Approximate Tensor Operations](https://mlsanity.com/paper/1805.08079/) (2018)
- [Minimal Effort Back Propagation for Convolutional Neural Networks](https://mlsanity.com/paper/1709.05804/) (2017)
- [Training Simplification and Model Simplification for Deep Learning: A Minimal Effort Back Propagation Method](https://mlsanity.com/paper/1711.06528/) (2017)
- [Structurally Sparsified Backward Propagation for Faster Long Short-Term Memory Training](https://mlsanity.com/paper/1806.00512/) (2018)
- [SEFR: A Fast Linear-Time Classifier for Ultra-Low Power Devices](https://mlsanity.com/paper/2006.04620/) (2020)

## Built on by (most cited first, 4 of 4)

- [On-device Online Learning and Semantic Management of TinyML Systems](https://mlsanity.com/paper/2405.07601/) (2024)
- [Advancing On-Device Neural Network Training with TinyPropv2: Dynamic, Sparse, and Efficient Backpropagation](https://mlsanity.com/paper/2409.07109/) (2024)
- [TActiLE: Tiny Active LEarning for wearable devices](https://mlsanity.com/paper/2505.01160/) (2025)
- [What changes after deployment? A survey on On-device Learning in TinyML](https://mlsanity.com/paper/2605.31226/) (2026)

## Abstract (arXiv)

Training deep neural networks using backpropagation is very memory and computationally intensive. This makes it difficult to run on-device learning or fine-tune neural networks on tiny, embedded devices such as low-power micro-controller units (MCUs). Sparse backpropagation algorithms try to reduce the computational load of on-device learning by training only a subset of the weights and biases. Existing approaches use a static number of weights to train. A poor choice of this so-called backpropagation ratio limits either the computational gain or can lead to severe accuracy losses. In this paper we present TinyProp, the first sparse backpropagation method that dynamically adapts the back-propagation ratio during on-device training for each training step. TinyProp induces a small calculation overhead to sort the elements of the gradient, which does not significantly impact the computational gains. TinyProp works particularly well on fine-tuning trained networks on MCUs, which is a typical use case for embedded applications. For typical datasets from three datasets MNIST, DCASE2020 and CIFAR10, we are 5 times faster compared to non-sparse training with an accuracy loss of on average 1%. On average, TinyProp is 2.9 times faster than existing, static sparse backpropagation algorithms and the accuracy loss is reduced on average by 6 % compared to a typical static setting of the back-propagation ratio.