# Dynamic Sparse Graph for Efficient Deep Learning

arXiv: https://arxiv.org/abs/1810.00859 · HTML: https://mlsanity.com/paper/1810.00859/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Oct 2018.

## Built on by (most cited first, 18 of 18)

- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformer](https://mlsanity.com/paper/2303.17605/) (2023)
- [Tiny Machine Learning: Progress and Futures](https://mlsanity.com/paper/2403.19076/) (2024)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications](https://mlsanity.com/paper/2204.11786/) (2022)
- [On-device Training: A First Overview on Existing Systems](https://mlsanity.com/paper/2212.00824/) (2022)
- [Accelerating Text-to-Image Editing via Cache-Enabled Sparse Diffusion Inference](https://mlsanity.com/paper/2305.17423/) (2023)
- [Enabling Resource-efficient AIoT System with Cross-level Optimization: A survey](https://mlsanity.com/paper/2309.15467/) (2023)
- [On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M Microcontrollers](https://mlsanity.com/paper/2407.10734/) (2024)
- [Data-Free Dynamic Compression of CNNs for Tractable Efficiency](https://mlsanity.com/paper/2309.17211/) (2023)
- [MERCURY: Accelerating DNN Training By Exploiting Input Similarity](https://mlsanity.com/paper/2110.14904/) (2021)
- [BitTrain: Sparse Bitmap Compression for Memory-Efficient Training on the Edge](https://mlsanity.com/paper/2110.15362/) (2021)
- [A Dimensionality Reduction Approach for Convolutional Neural Networks](https://mlsanity.com/paper/2110.09163/) (2021)
- [Exploiting Activation based Gradient Output Sparsity to Accelerate Backpropagation in CNNs](https://mlsanity.com/paper/2109.07710/) (2021)
- [Breaking On-device Training Memory Wall: A Systematic Survey](https://mlsanity.com/paper/2306.10388/) (2023)
- [Enhance DNN Adversarial Robustness and Efficiency via Injecting Noise to Non-Essential Neurons](https://mlsanity.com/paper/2402.04325/) (2024)
- [Adaptive Pruning for Large Language Models with Structural Importance Awareness](https://mlsanity.com/paper/2412.15127/) (2024)
- [HASTE: A Framework for Training-Free, Dynamic, and Steerable Compression of Pre-Trained Convolutional Neural Networks](https://mlsanity.com/paper/2606.30516/) (2026)

## Abstract (arXiv)

We propose to execute deep neural networks (DNNs) with dynamic and sparse graph (DSG) structure for compressive memory and accelerative execution during both training and inference. The great success of DNNs motivates the pursuing of lightweight models for the deployment onto embedded devices. However, most of the previous studies optimize for inference while neglect training or even complicate it. Training is far more intractable, since (i) the neurons dominate the memory cost rather than the weights in inference; (ii) the dynamic activation makes previous sparse acceleration via one-off optimization on fixed weight invalid; (iii) batch normalization (BN) is critical for maintaining accuracy while its activation reorganization damages the sparsity. To address these issues, DSG activates only a small amount of neurons with high selectivity at each iteration via a dimension-reduction search (DRS) and obtains the BN compatibility via a double-mask selection (DMS). Experiments show significant memory saving (1.7-4.5x) and operation reduction (2.3-4.4x) with little accuracy loss on various benchmarks.