# Advancing On-Device Neural Network Training with TinyPropv2: Dynamic, Sparse, and Efficient Backpropagation

arXiv: https://arxiv.org/abs/2409.07109 · HTML: https://mlsanity.com/paper/2409.07109/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 11 Sep 2024.

## Builds on (most cited first, 17 of 17)

- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [TinyOL: TinyML with Online-Learning on Microcontrollers](https://mlsanity.com/paper/2103.08295/) (2021)
- [meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting](https://mlsanity.com/paper/1706.06197/) (2017)
- [TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning](https://mlsanity.com/paper/2007.11622/) (2020)
- [PockEngine: Sparse and Efficient Fine-tuning in a Pocket](https://mlsanity.com/paper/2310.17752/) (2023)
- [Sparse Backpropagation for MoE Training](https://mlsanity.com/paper/2310.00811/) (2023)
- [Efficient On-device Training via Gradient Filtering](https://mlsanity.com/paper/2301.00330/) (2023)
- [SparseProp: Efficient Sparse Backpropagation for Faster Training of Neural Networks](https://mlsanity.com/paper/2302.04852/) (2023)
- [Minimal Effort Back Propagation for Convolutional Neural Networks](https://mlsanity.com/paper/1709.05804/) (2017)
- [Training Simplification and Model Simplification for Deep Learning: A Minimal Effort Back Propagation Method](https://mlsanity.com/paper/1711.06528/) (2017)
- [Structurally Sparsified Backward Propagation for Faster Long Short-Term Memory Training](https://mlsanity.com/paper/1806.00512/) (2018)
- [SEFR: A Fast Linear-Time Classifier for Ultra-Low Power Devices](https://mlsanity.com/paper/2006.04620/) (2020)
- [TinyProp -- Adaptive Sparse Backpropagation for Efficient TinyML On-device Learning](https://mlsanity.com/paper/2308.09201/) (2023)

## Built on by (most cited first, 3 of 3)

- [DRIP: DRop unImportant data Points -- Enhancing Machine Learning Efficiency with Grad-CAM-Based Real-Time Data Prioritization for On-Device Training](https://mlsanity.com/paper/2504.08364/) (2025)
- [What changes after deployment? A survey on On-device Learning in TinyML](https://mlsanity.com/paper/2605.31226/) (2026)
- [Toward a Modular Architecture for Embedded AI Agent Systems at the Edge](https://mlsanity.com/paper/2606.02862/) (2026)

## Abstract (arXiv)

This study introduces TinyPropv2, an innovative algorithm optimized for on-device learning in deep neural networks, specifically designed for low-power microcontroller units. TinyPropv2 refines sparse backpropagation by dynamically adjusting the level of sparsity, including the ability to selectively skip training steps. This feature significantly lowers computational effort without substantially compromising accuracy. Our comprehensive evaluation across diverse datasets CIFAR 10, CIFAR100, Flower, Food, Speech Command, MNIST, HAR, and DCASE2020 reveals that TinyPropv2 achieves near-parity with full training methods, with an average accuracy drop of only around 1 percent in most cases. For instance, against full training, TinyPropv2's accuracy drop is minimal, for example, only 0.82 percent on CIFAR 10 and 1.07 percent on CIFAR100. In terms of computational effort, TinyPropv2 shows a marked reduction, requiring as little as 10 percent of the computational effort needed for full training in some scenarios, and consistently outperforms other sparse training methodologies. These findings underscore TinyPropv2's capacity to efficiently manage computational resources while maintaining high accuracy, positioning it as an advantageous solution for advanced embedded device applications in the IoT ecosystem.