# Dynamic Gradient Sparse Update for Edge Training

arXiv: https://arxiv.org/abs/2503.17959 · HTML: https://mlsanity.com/paper/2503.17959/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 18 references that are in the map.
- First posted on arXiv 23 Mar 2025.

## Builds on (most cited first, 15 of 15)

- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [DepGraph: Towards Any Structural Pruning](https://mlsanity.com/paper/2301.12900/) (2023)
- [Unlearnable Examples: Making Personal Data Unexploitable](https://mlsanity.com/paper/2101.04898/) (2021)
- [On-Device Training Under 256KB Memory](https://mlsanity.com/paper/2206.15472/) (2022)
- [Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs](https://mlsanity.com/paper/2003.00152/) (2020)
- [Revisiting Small Batch Training for Deep Neural Networks](https://mlsanity.com/paper/1804.07612/) (2018)
- [PatDNN: Achieving Real-Time DNN Execution on Mobile Devices with Pattern-based Weight Pruning](https://mlsanity.com/paper/2001.00138/) (2020)
- [Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks](https://mlsanity.com/paper/1911.09737/) (2019)
- [TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning](https://mlsanity.com/paper/2007.11622/) (2020)
- [VanillaKD: Revisit the Power of Vanilla Knowledge Distillation from Small Scale to Large Scale](https://mlsanity.com/paper/2305.15781/) (2023)
- [Zebra: Memory Bandwidth Reduction for CNN Accelerators With Zero Block Regularization of Activation Maps](https://mlsanity.com/paper/2205.00779/) (2022)
- [NoMorelization: Building Normalizer-Free Models from a Sample's Perspective](https://mlsanity.com/paper/2210.06932/) (2022)

## Abstract (arXiv)

Training on edge devices enables personalized model fine-tuning to enhance real-world performance and maintain data privacy. However, the gradient computation for backpropagation in the training requires significant memory buffers to store intermediate features and compute losses. This is unacceptable for memory-constrained edge devices such as microcontrollers. To tackle this issue, we propose a training acceleration method using dynamic gradient sparse updates. This method updates the important channels and layers only and skips gradient computation for the less important channels and layers to reduce memory usage for each update iteration. In addition, the channel selection is dynamic for different iterations to traverse most of the parameters in the update layers along the time dimension for better performance. The experimental result shows that the proposed method enables an ImageNet pre-trained MobileNetV2 trained on CIFAR-10 to achieve an accuracy of 85.77% while updating only 2% of convolution weights within 256KB on-chip memory. This results in a remarkable 98% reduction in feature memory usage compared to dense model training.