# Exploring the Regularity of Sparse Structure in Convolutional Neural Networks

arXiv: https://arxiv.org/abs/1705.08922 · HTML: https://mlsanity.com/paper/1705.08922/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2017.

## Built on by (most cited first, 30 of 37)

- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021)
- [Hessian-Aware Pruning and Optimal Neural Implant](https://mlsanity.com/paper/2101.08940/) (2021)
- [Coarsening the Granularity: Towards Structurally Sparse Lottery Tickets](https://mlsanity.com/paper/2202.04736/) (2022)
- [Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Cases](https://mlsanity.com/paper/2301.12017/) (2023)
- [Compiler-Aware Neural Architecture Search for On-Mobile Real-time Super-Resolution](https://mlsanity.com/paper/2207.12577/) (2022)
- [Towards Sparsification of Graph Neural Networks](https://mlsanity.com/paper/2209.04766/) (2022)
- [Efficient Quantized Sparse Matrix Operations on Tensor Cores](https://mlsanity.com/paper/2209.06979/) (2022)
- [Towards Artificial General Intelligence (AGI) in the Internet of Things (IoT): Opportunities and Challenges](https://mlsanity.com/paper/2309.07438/) (2023)
- [JaxPruner: A concise library for sparsity research](https://mlsanity.com/paper/2304.14082/) (2023)
- [Lottery Jackpots Exist in Pre-trained Models](https://mlsanity.com/paper/2104.08700/) (2021)
- [Dancing along Battery: Enabling Transformer with Run-time Reconfigurability on Mobile Devices](https://mlsanity.com/paper/2102.06336/) (2021)
- [Improving the Accuracy of Early Exits in Multi-Exit Architectures via Curriculum Learning](https://mlsanity.com/paper/2104.10461/) (2021)
- [Training Acceleration of Low-Rank Decomposed Networks using Sequential Freezing and Rank Quantization](https://mlsanity.com/paper/2309.03824/) (2023)
- [Speeding up Resnet Architecture with Layers Targeted Low Rank Decomposition](https://mlsanity.com/paper/2309.12412/) (2023)
- [No Free Prune: Information-Theoretic Barriers to Pruning at Initialization](https://mlsanity.com/paper/2402.01089/) (2024)
- [LilNetX: Lightweight Networks with EXtreme Model Compression and Structured Sparsification](https://mlsanity.com/paper/2204.02965/) (2022)
- [Mixed-Precision Neural Networks: A Survey](https://mlsanity.com/paper/2208.06064/) (2022)
- [Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognition](https://mlsanity.com/paper/2112.11438/) (2021)
- [Pruning for Feature-Preserving Circuits in CNNs](https://mlsanity.com/paper/2206.01627/) (2022)
- [RingCNN: Exploiting Algebraically-Sparse Ring Tensors for Energy-Efficient CNN-Based Computational Imaging](https://mlsanity.com/paper/2104.09056/) (2021)
- [Entropy Induced Pruning Framework for Convolutional Neural Networks](https://mlsanity.com/paper/2208.06660/) (2022)
- [PruMUX: Augmenting Data Multiplexing with Model Compression](https://mlsanity.com/paper/2305.14706/) (2023)
- [S2Engine: A Novel Systolic Architecture for Sparse Convolutional Neural Networks](https://mlsanity.com/paper/2106.07894/) (2021)
- [BRDS: An FPGA-based LSTM Accelerator with Row-Balanced Dual-Ratio Sparsification](https://mlsanity.com/paper/2101.02667/) (2021)
- [Load-balanced Gather-scatter Patterns for Sparse Deep Neural Networks](https://mlsanity.com/paper/2112.10898/) (2021)
- [EcoFlow: Efficient Convolutional Dataflows for Low-Power Neural Network Accelerators](https://mlsanity.com/paper/2202.02310/) (2022)
- [Accelerating the Low-Rank Decomposed Models](https://mlsanity.com/paper/2407.20266/) (2024)
- [SmartDeal: Re-Modeling Deep Network Weights for Efficient Inference and Training](https://mlsanity.com/paper/2101.01163/) (2021)
- [Cascade Weight Shedding in Deep Neural Networks: Benefits and Pitfalls for Network Pruning](https://mlsanity.com/paper/2103.10629/) (2021)

## Abstract (arXiv)

Sparsity helps reduce the computational complexity of deep neural networks by skipping zeros. Taking advantage of sparsity is listed as a high priority in next generation DNN accelerators such as TPU. The structure of sparsity, i.e., the granularity of pruning, affects the efficiency of hardware accelerator design as well as the prediction accuracy. Coarse-grained pruning creates regular sparsity patterns, making it more amenable for hardware acceleration but more challenging to maintain the same accuracy. In this paper we quantitatively measure the trade-off between sparsity regularity and prediction accuracy, providing insights in how to maintain accuracy while having more a more structured sparsity pattern. Our experimental results show that coarse-grained pruning can achieve a sparsity ratio similar to unstructured pruning without loss of accuracy. Moreover, due to the index saving effect, coarse-grained pruning is able to obtain a better compression ratio than fine-grained sparsity at the same accuracy threshold. Based on the recent sparse convolutional neural network accelerator (SCNN), our experiments further demonstrate that coarse-grained sparsity saves about 2x the memory references compared to fine-grained sparsity. Since memory reference is more than two orders of magnitude more expensive than arithmetic operations, the regularity of sparse structure leads to more efficient hardware design.