# Efficient N:M Sparse DNN Training Using Algorithm, Architecture, and Dataflow Co-Design

arXiv: https://arxiv.org/abs/2309.13015 · HTML: https://mlsanity.com/paper/2309.13015/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 70 references that are in the map.
- First posted on arXiv 22 Sep 2023.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019)
- [Accelerating Sparse Deep Neural Networks](https://mlsanity.com/paper/2104.08378/) (2021)
- [Low-bit Quantization of Neural Networks for Efficient Inference](https://mlsanity.com/paper/1902.06822/) (2019)
- [Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch](https://mlsanity.com/paper/2102.04010/) (2021)
- [PatDNN: Achieving Real-Time DNN Execution on Mobile Devices with Pattern-based Weight Pruning](https://mlsanity.com/paper/2001.00138/) (2020)
- [SPDY: Accurate Pruning with Speedup Guarantees](https://mlsanity.com/paper/2201.13096/) (2022)
- [FAST: DNN Training Under Variable Precision Block Floating Point with Stochastic Rounding](https://mlsanity.com/paper/2110.15456/) (2021)
- [Adaptable Butterfly Accelerator for Attention-based NNs via Hardware and Algorithm Co-design](https://mlsanity.com/paper/2209.09570/) (2022)
- [An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers](https://mlsanity.com/paper/2208.06118/) (2022)
- [Growing Efficient Deep Networks by Structured Continuous Sparsification](https://mlsanity.com/paper/2007.15353/) (2020)
- [Enabling On-Device CNN Training by Self-Supervised Instance Filtering and Error Map Pruning](https://mlsanity.com/paper/2007.03213/) (2020)
- [1xN Pattern for Pruning Convolutional Neural Networks](https://mlsanity.com/paper/2105.14713/) (2021)
- [Pre-defined Sparsity for Low-Complexity Convolutional Neural Networks](https://mlsanity.com/paper/2001.10710/) (2020)
- [Dynamic N:M Fine-grained Structured Sparse Attention Mechanism](https://mlsanity.com/paper/2203.00091/) (2022)
- [SCANN: Synthesis of Compact and Accurate Neural Networks](https://mlsanity.com/paper/1904.09090/) (2019)
- [SparseTrain: Exploiting Dataflow Sparsity for Efficient Convolutional Neural Networks Training](https://mlsanity.com/paper/2007.13595/) (2020)
- [EF-Train: Enable Efficient On-device CNN Training on FPGA Through Data Reshaping for Online Adaptation or Personalization](https://mlsanity.com/paper/2202.10935/) (2022)
- [Accelerating DNN Training with Structured Data Gradient Pruning](https://mlsanity.com/paper/2202.00774/) (2022)
- [Automatic Compiler Based FPGA Accelerator for CNN Training](https://mlsanity.com/paper/1908.06724/) (2019)
- [Approximate Random Dropout](https://mlsanity.com/paper/1805.08939/) (2018)
- [FeCaffe: FPGA-enabled Caffe with OpenCL for Deep Learning Training and Inference on Intel Stratix 10](https://mlsanity.com/paper/1911.08905/) (2019)

## Built on by (most cited first, 5 of 5)

- [Wanda++: Pruning Large Language Models via Regional Gradients](https://mlsanity.com/paper/2503.04992/) (2025)
- [Ultra Memory-Efficient On-FPGA Training of Transformers via Tensor-Compressed Optimization](https://mlsanity.com/paper/2501.06663/) (2025)
- [An Efficient Training Algorithm for Models with Block-wise Sparsity](https://mlsanity.com/paper/2503.21928/) (2025)
- [TSENOR: Highly-Efficient Algorithm for Finding Transposable N:M Sparse Masks](https://mlsanity.com/paper/2505.23949/) (2025)
- [Comprehensive Design Space Exploration for Tensorized Neural Network Hardware Accelerators](https://mlsanity.com/paper/2511.17971/) (2025)

## Abstract (arXiv)

Sparse training is one of the promising techniques to reduce the computational cost of DNNs while retaining high accuracy. In particular, N:M fine-grained structured sparsity, where only N out of consecutive M elements can be nonzero, has attracted attention due to its hardware-friendly pattern and capability of achieving a high sparse ratio. However, the potential to accelerate N:M sparse DNN training has not been fully exploited, and there is a lack of efficient hardware supporting N:M sparse training. To tackle these challenges, this paper presents a computation-efficient training scheme for N:M sparse DNNs using algorithm, architecture, and dataflow co-design. At the algorithm level, a bidirectional weight pruning method, dubbed BDWP, is proposed to leverage the N:M sparsity of weights during both forward and backward passes of DNN training, which can significantly reduce the computational cost while maintaining model accuracy. At the architecture level, a sparse accelerator for DNN training, namely SAT, is developed to neatly support both the regular dense operations and the computation-efficient N:M sparse operations. At the dataflow level, multiple optimization methods ranging from interleave mapping, pre-generation of N:M sparse weights, and offline scheduling, are proposed to boost the computational efficiency of SAT. Finally, the effectiveness of our training scheme is evaluated on a Xilinx VCU1525 FPGA card using various DNN models and datasets. Experimental results show the SAT accelerator with the BDWP sparse training method under 2:8 sparse ratio achieves an average speedup of 1.75x over that with the dense training, accompanied by a negligible accuracy loss of 0.56% on average. Furthermore, our proposed training scheme significantly improves the training throughput by 2.97~25.22x and the energy efficiency by 1.36~3.58x over prior FPGA-based accelerators.