# SparseTrain: Exploiting Dataflow Sparsity for Efficient Convolutional Neural Networks Training

arXiv: https://arxiv.org/abs/2007.13595 · HTML: https://mlsanity.com/paper/2007.13595/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jul 2020.

## Built on by (most cited first, 10 of 10)

- [Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications](https://mlsanity.com/paper/2204.11786/) (2022)
- [Enabling Resource-efficient AIoT System with Cross-level Optimization: A survey](https://mlsanity.com/paper/2309.15467/) (2023)
- [Efficient N:M Sparse DNN Training Using Algorithm, Architecture, and Dataflow Co-Design](https://mlsanity.com/paper/2309.13015/) (2023)
- [TinyFormer: Efficient Transformer Design and Deployment on Tiny Devices](https://mlsanity.com/paper/2311.01759/) (2023)
- [Exploiting Activation based Gradient Output Sparsity to Accelerate Backpropagation in CNNs](https://mlsanity.com/paper/2109.07710/) (2021)
- [LearningGroup: A Real-Time Sparse Training on FPGA via Learnable Weight Grouping for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2210.16624/) (2022)
- [Signed Binary Weight Networks](https://mlsanity.com/paper/2211.13838/) (2022)
- [DDC-PIM: Efficient Algorithm/Architecture Co-design for Doubling Data Capacity of SRAM-based Processing-In-Memory](https://mlsanity.com/paper/2310.20424/) (2023)
- [Block-Wise Dynamic-Precision Neural Network Training Acceleration via Online Quantization Sensitivity Analytics](https://mlsanity.com/paper/2210.17047/) (2022)
- [PLUM: Improving Inference Efficiency By Leveraging Repetition-Sparsity Trade-Off](https://mlsanity.com/paper/2312.01581/) (2023)

## Abstract (arXiv)

Training Convolutional Neural Networks (CNNs) usually requires a large number of computational resources. In this paper, textitSparseTrain is proposed to accelerate CNN training by fully exploiting the sparsity. It mainly involves three levels of innovations: activation gradients pruning algorithm, sparse training dataflow, and accelerator architecture. By applying a stochastic pruning algorithm on each layer, the sparsity of back-propagation gradients can be increased dramatically without degrading training accuracy and convergence rate. Moreover, to utilize both textitnatural sparsity (resulted from ReLU or Pooling layers) and textitartificial sparsity (brought by pruning algorithm), a sparse-aware architecture is proposed for training acceleration. This architecture supports forward and back-propagation of CNN by adopting 1-Dimensional convolution dataflow. We have built %a simple compiler to map CNNs topology onto textitSparseTrain, and a cycle-accurate architecture simulator to evaluate the performance and efficiency based on the synthesized design with 14nm FinFET technologies. Evaluation results on AlexNet/ResNet show that textitSparseTrain could achieve about 2.7 × speedup and 2.2 × energy efficiency improvement on average compared with the original training process.