# SparseRT: Accelerating Unstructured Sparsity on GPUs for Deep Learning Inference

arXiv: https://arxiv.org/abs/2008.11849 · HTML: https://mlsanity.com/paper/2008.11849/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Aug 2020.

## Built on by (most cited first, 30 of 30)

- [SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks](https://mlsanity.com/paper/2402.09025/) (2024)
- [ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsity](https://mlsanity.com/paper/2208.02507/) (2022)
- [Learning Best Combination for Efficient N:M Sparsity](https://mlsanity.com/paper/2206.06662/) (2022)
- [PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformation](https://mlsanity.com/paper/2301.10936/) (2023)
- [SparseDNN: Fast Sparse Deep Learning Inference on CPUs](https://mlsanity.com/paper/2101.07948/) (2021)
- [SparseProp: Efficient Sparse Backpropagation for Faster Training of Neural Networks](https://mlsanity.com/paper/2302.04852/) (2023)
- [Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey](https://mlsanity.com/paper/2205.08099/) (2022)
- [Sparse Distributed Memory is a Continual Learner](https://mlsanity.com/paper/2303.11934/) (2023)
- [CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning](https://mlsanity.com/paper/2509.22075/) (2025)
- [Neuralink: Fast LLM Inference on Smartphones with Neuron Co-Activation Linking](https://mlsanity.com/paper/2410.19274/) (2024)
- [Transformers meet Stochastic Block Models: Attention with Data-Adaptive Sparsity and Cost](https://mlsanity.com/paper/2210.15541/) (2022)
- [Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models](https://mlsanity.com/paper/2311.04386/) (2023)
- [PopSparse: Accelerated block sparse matrix multiplication on IPU](https://mlsanity.com/paper/2303.16999/) (2023)
- [Low-Bitwidth Floating Point Quantization for Efficient High-Quality Diffusion Models](https://mlsanity.com/paper/2408.06995/) (2024)
- [DReSS: Data-driven Regularized Structured Streamlining for Large Language Models](https://mlsanity.com/paper/2501.17905/) (2025)
- [Hardware/Software Co-Design of RISC-V Extensions for Accelerating Sparse DNNs on FPGAs](https://mlsanity.com/paper/2504.19659/) (2025)
- [End-to-End Supermask Pruning: Learning to Prune Image Captioning Models](https://mlsanity.com/paper/2110.03298/) (2021)
- [An Efficient Sparse Inference Software Accelerator for Transformer-based Language Models on CPUs](https://mlsanity.com/paper/2306.16601/) (2023)
- [Edge-InversionNet: Enabling Efficient Inference of InversionNet on Edge Devices](https://mlsanity.com/paper/2310.09667/) (2023)
- [EvoP: Robust LLM Inference via Evolutionary Pruning](https://mlsanity.com/paper/2502.14910/) (2025)
- [Gradient-based Weight Density Balancing for Robust Dynamic Sparse Training](https://mlsanity.com/paper/2210.14012/) (2022)
- [STen: Productive and Efficient Sparsity in PyTorch](https://mlsanity.com/paper/2304.07613/) (2023)
- [Spatial Re-parameterization for N:M Sparsity](https://mlsanity.com/paper/2306.05612/) (2023)
- [The Sparsity Roofline: Understanding the Hardware Limits of Sparse Neural Networks](https://mlsanity.com/paper/2310.00496/) (2023)
- [GM-Skip: Metric-Guided Transformer Block Skipping for Efficient Vision-Language Models](https://mlsanity.com/paper/2508.18227/) (2025)
- [A Generalization of Continuous Relaxation in Structured Pruning](https://mlsanity.com/paper/2308.14605/) (2023)
- [FiRST: Finetuning Router-Selective Transformers for Input-Adaptive Latency Reduction](https://mlsanity.com/paper/2410.12513/) (2024)
- [Learning Compatible Multi-Prize Subnetworks for Asymmetric Retrieval](https://mlsanity.com/paper/2504.11879/) (2025)
- [Performance and Complexity Trade-off Optimization of Speech Models During Training](https://mlsanity.com/paper/2601.13704/) (2026)
- [A Comparative Study of CNN Optimization Methods for Edge AI: Exploring the Role of Early Exits](https://mlsanity.com/paper/2604.14789/) (2026)

## Abstract (arXiv)

In recent years, there has been a flurry of research in deep neural network pruning and compression. Early approaches prune weights individually. However, it is difficult to take advantage of the resulting unstructured sparsity patterns on modern hardware like GPUs. As a result, pruning strategies which impose sparsity structures in the weights have become more popular. However,these structured pruning approaches typically lead to higher losses in accuracy than unstructured pruning. In this paper, we present SparseRT, a code generator that leverage unstructured sparsity to accelerate sparse linear algebra operations in deep learning inference on GPUs. For 1x1 convolutions and fully connected layers, we demonstrate geometric mean of speedups of 3.4x over the equivalent dense computation at 90% sparsity and 5.4x at 95% sparsity when evaluated on hundreds of test cases in deep learning. For sparse 3x3 convolutions, we show speedups of over 5x on use cases in ResNet-50.