# Balanced Sparsity for Efficient DNN Inference on GPU

arXiv: https://arxiv.org/abs/1811.00206 · HTML: https://mlsanity.com/paper/1811.00206/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2018.

## Built on by (most cited first, 27 of 27)

- [Accelerating Sparse Deep Neural Networks](https://mlsanity.com/paper/2104.08378/) (2021)
- [MegaBlocks: Efficient Sparse Training with Mixture-of-Experts](https://mlsanity.com/paper/2211.15841/) (2022)
- [FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout](https://mlsanity.com/paper/2102.13451/) (2021)
- [Dual-side Sparse Tensor Core](https://mlsanity.com/paper/2105.09564/) (2021)
- [Effective Interplay between Sparsity and Quantization: From Theory to Practice](https://mlsanity.com/paper/2405.20935/) (2024)
- [Multi-Exit Semantic Segmentation Networks](https://mlsanity.com/paper/2106.03527/) (2021)
- [Progressive Gradient Flow for Robust N:M Sparsity Training in Transformers](https://mlsanity.com/paper/2402.04744/) (2024)
- [Sparse maximal update parameterization: A holistic approach to sparse training dynamics](https://mlsanity.com/paper/2405.15743/) (2024)
- [Two Sparsities Are Better Than One: Unlocking the Performance Benefits of Sparse-Sparse Networks](https://mlsanity.com/paper/2112.13896/) (2021)
- [E-Sparse: Boosting the Large Language Model Inference through Entropy-based N:M Sparsity](https://mlsanity.com/paper/2310.15929/) (2023)
- [SparseDNN: Fast Sparse Deep Learning Inference on CPUs](https://mlsanity.com/paper/2101.07948/) (2021)
- [Attentive Fine-Grained Structured Sparsity for Image Restoration](https://mlsanity.com/paper/2204.12266/) (2022)
- [Training Recipe for N:M Structured Sparsity with Decaying Pruning Mask](https://mlsanity.com/paper/2209.07617/) (2022)
- [Low-Rank Prune-And-Factorize for Language Model Compression](https://mlsanity.com/paper/2306.14152/) (2023)
- [The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models](https://mlsanity.com/paper/2312.00960/) (2023)
- [Cut Less, Fold More: Model Compression through the Lens of Projection Geometry](https://mlsanity.com/paper/2602.18116/) (2026)
- [PopSparse: Accelerated block sparse matrix multiplication on IPU](https://mlsanity.com/paper/2303.16999/) (2023)
- [Pruning Compact ConvNets for Efficient Inference](https://mlsanity.com/paper/2301.04502/) (2023)
- [Graph Conditioned Sparse-Attention for Improved Source Code Understanding](https://mlsanity.com/paper/2112.00663/) (2021)
- [Physics-aware Roughness Optimization for Diffractive Optical Neural Networks](https://mlsanity.com/paper/2304.01500/) (2023)
- [STen: Productive and Efficient Sparsity in PyTorch](https://mlsanity.com/paper/2304.07613/) (2023)
- [Cascade Weight Shedding in Deep Neural Networks: Benefits and Pitfalls for Network Pruning](https://mlsanity.com/paper/2103.10629/) (2021)
- [Tight Compression: Compressing CNN Through Fine-Grained Pruning and Weight Permutation for Efficient Implementation](https://mlsanity.com/paper/2104.01303/) (2021)
- [A Generalization of Continuous Relaxation in Structured Pruning](https://mlsanity.com/paper/2308.14605/) (2023)
- [NVR: Vector Runahead on NPUs for Sparse Memory Access](https://mlsanity.com/paper/2502.13873/) (2025)
- [Decoding for Punctured Convolutional and Turbo Codes: A Deep Learning Solution for Protocols Compliance](https://mlsanity.com/paper/2502.15475/) (2025)
- [A Comparative Study of CNN Optimization Methods for Edge AI: Exploring the Role of Early Exits](https://mlsanity.com/paper/2604.14789/) (2026)

## Abstract (arXiv)

In trained deep neural networks, unstructured pruning can reduce redundant weights to lower storage cost. However, it requires the customization of hardwares to speed up practical inference. Another trend accelerates sparse model inference on general-purpose hardwares by adopting coarse-grained sparsity to prune or regularize consecutive weights for efficient computation. But this method often sacrifices model accuracy. In this paper, we propose a novel fine-grained sparsity approach, balanced sparsity, to achieve high model accuracy with commercial hardwares efficiently. Our approach adapts to high parallelism property of GPU, showing incredible potential for sparsity in the widely deployment of deep learning services. Experiment results show that balanced sparsity achieves up to 3.1x practical speedup for model inference on GPU, while retains the same high model accuracy as fine-grained sparsity.