# SparseDNN: Fast Sparse Deep Learning Inference on CPUs

arXiv: https://arxiv.org/abs/2101.07948 · HTML: https://mlsanity.com/paper/2101.07948/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 39 references that are in the map.
- First posted on arXiv 20 Jan 2021.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)
- [EIE: Efficient Inference Engine on Compressed Deep Neural Network](https://mlsanity.com/paper/1602.01528/) (2016)
- [Variational Dropout Sparsifies Deep Neural Networks](https://mlsanity.com/paper/1701.05369/) (2017)
- [cuDNN: Efficient Primitives for Deep Learning](https://mlsanity.com/paper/1410.0759/) (2014)
- [Structured Pruning of Large Language Models](https://mlsanity.com/paper/1910.04732/) (2019)
- [Sparse GPU Kernels for Deep Learning](https://mlsanity.com/paper/2006.10901/) (2020)
- [Fast Sparse ConvNets](https://mlsanity.com/paper/1911.09723/) (2019)
- [MNN: A Universal and Efficient Inference Engine](https://mlsanity.com/paper/2002.12418/) (2020)
- [Faster CNNs with Direct Sparse Convolutions and Guided Pruning](https://mlsanity.com/paper/1608.01409/) (2016)
- [SparseRT: Accelerating Unstructured Sparsity on GPUs for Deep Learning Inference](https://mlsanity.com/paper/2008.11849/) (2020)
- [Optimizing CNN Model Inference on CPUs](https://mlsanity.com/paper/1809.02697/) (2018)
- [Balanced Sparsity for Efficient DNN Inference on GPU](https://mlsanity.com/paper/1811.00206/) (2018)
- [Design Principles for Sparse Matrix Multiplication on the GPU](https://mlsanity.com/paper/1803.08601/) (2018)
- [Neural Network Compression Framework for fast model inference](https://mlsanity.com/paper/2002.08679/) (2020)
- [Escoin: Efficient Sparse Convolutional Neural Network Inference on GPUs](https://mlsanity.com/paper/1802.10280/) (2018)
- [High-Performance Deep Learning via a Single Building Block](https://mlsanity.com/paper/1906.06440/) (2019)

## Built on by (most cited first, 13 of 13)

- [ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsity](https://mlsanity.com/paper/2208.02507/) (2022)
- [Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications](https://mlsanity.com/paper/2204.11786/) (2022)
- [SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models](https://mlsanity.com/paper/2303.10464/) (2023)
- [Two Sparsities Are Better Than One: Unlocking the Performance Benefits of Sparse-Sparse Networks](https://mlsanity.com/paper/2112.13896/) (2021)
- [SNP: Structured Neuron-level Pruning to Preserve Attention Scores](https://mlsanity.com/paper/2404.11630/) (2024)
- [Signal Collapse in One-Shot Pruning: When Sparse Models Fail to Distinguish Neural Representations](https://mlsanity.com/paper/2502.15790/) (2025)
- [DRIVE: Dual Gradient-Based Rapid Iterative Pruning](https://mlsanity.com/paper/2404.03687/) (2024)
- [Fast Tree-Field Integrators: From Low Displacement Rank to Topological Transformers](https://mlsanity.com/paper/2406.15881/) (2024)
- [An Efficient Sparse Inference Software Accelerator for Transformer-based Language Models on CPUs](https://mlsanity.com/paper/2306.16601/) (2023)
- [Lightweight Software Kernels and Hardware Extensions for Efficient Sparse Deep Neural Networks on Microcontrollers](https://mlsanity.com/paper/2503.06183/) (2025)
- [Rosko: Row Skipping Outer Products for Sparse Matrix Multiplication Kernels](https://mlsanity.com/paper/2307.03930/) (2023)
- [Sculpting Efficiency: Pruning Medical Imaging Models for On-Device Inference](https://mlsanity.com/paper/2309.05090/) (2023)
- [MediSwift: Efficient Sparse Pre-trained Biomedical Language Models](https://mlsanity.com/paper/2403.00952/) (2024)

## Abstract (arXiv)

The last few years have seen gigantic leaps in algorithms and systems to support efficient deep learning inference. Pruning and quantization algorithms can now consistently compress neural networks by an order of magnitude. For a compressed neural network, a multitude of inference frameworks have been designed to maximize the performance of the target hardware. While we find mature support for quantized neural networks in production frameworks such as OpenVINO and MNN, support for pruned sparse neural networks is still lacking. To tackle this challenge, we present SparseDNN, a sparse deep learning inference engine targeting CPUs. We present both kernel-level optimizations with a sparse code generator to accelerate sparse operators and novel network-level optimizations catering to sparse networks. We show that our sparse code generator can achieve significant speedups over state-of-the-art sparse and dense libraries. On end-to-end benchmarks such as Huggingface pruneBERT, SparseDNN achieves up to 5x throughput improvement over dense inference with state-of-the-art OpenVINO. Open source library at: https://github.com/marsupialtail/sparsednn.