# Accelerating Sparse DNNs Based on Tiled GEMM

arXiv: https://arxiv.org/abs/2402.10876 · HTML: https://mlsanity.com/paper/2402.10876/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Feb 2024.

## Built on by (most cited first, 8 of 8)

- [A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models](https://mlsanity.com/paper/2410.07265/) (2024)
- [M-ANT: Efficient Low-bit Group Quantization for LLMs via Mathematically Adaptive Numerical Type](https://mlsanity.com/paper/2502.18755/) (2025)
- [vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving](https://mlsanity.com/paper/2407.15309/) (2024)
- [Ecco: Improving Memory Bandwidth and Capacity for LLMs via Entropy-aware Cache Compression](https://mlsanity.com/paper/2505.06901/) (2025)
- [M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization](https://mlsanity.com/paper/2601.19213/) (2026)
- [Phi: Leveraging Pattern-based Hierarchical Sparsity for High-Efficiency Spiking Neural Networks](https://mlsanity.com/paper/2505.10909/) (2025)
- [Mitigating scalability challenges in LUT-based neural networks via pruning optimisations](https://mlsanity.com/paper/2407.02362/) (2024)
- [FractalCloud: A Fractal-Inspired Architecture for Efficient Large-Scale Point Cloud Processing](https://mlsanity.com/paper/2511.07665/) (2025)

## Abstract (arXiv)

Network pruning can reduce the computation cost of deep neural network (DNN) models. However, sparse models often produce randomly-distributed weights to maintain accuracy, leading to irregular computations. Consequently, unstructured sparse models cannot achieve meaningful speedup on commodity hardware built for dense matrix computations. Accelerators are usually modified or designed with structured sparsity-optimized architectures for exploiting sparsity. For example, the Ampere architecture introduces a sparse tensor core, which adopts the 2:4 sparsity pattern. We propose a pruning method that builds upon the insight that matrix multiplication generally breaks the large matrix into multiple smaller tiles for parallel execution. We present the tile-wise sparsity pattern, which maintains a structured sparsity pattern at the tile level for efficient execution but allows for irregular pruning at the global scale to maintain high accuracy. In addition, the tile-wise sparsity is implemented at the global memory level, and the 2:4 sparsity executes at the register level inside the sparse tensor core. We can combine these two patterns into a tile-vector-wise (TVW) sparsity pattern to explore more fine-grained sparsity and further accelerate the sparse DNN models. We evaluate the TVW on the GPU, achieving averages of 1.85×, 2.75×, and 22.18× speedups over the dense model, block sparsity, and unstructured sparsity.