Data to 7 Oct 2026

Sparse GPU Kernels for Deep Learning

arXiv 2006.10901 ↗First posted 18 Jun 2020cs.LG cs.DC stat.MLcited by 138

Not on the map. Why?

Abstract (arXiv)

Scientific workloads have traditionally exploited high levels of sparsity to accelerate computation and reduce memory requirements. While deep neural networks can be made sparse, achieving practical speedups on GPUs is difficult because these applications have relatively moderate levels of sparsity that are not sufficient for existing sparse kernels to outperform their dense counterparts. In this work, we study sparse matrices from deep learning applications and identify favorable properties that can be exploited to accelerate computation. Based on these insights, we develop high-performance GPU kernels for two sparse matrix operations widely applicable in neural networks: sparse matrix-dense matrix multiplication and sampled dense-dense matrix multiplication. Our kernels reach 27% of single-precision peak on Nvidia V100 GPUs. Using our kernels, we demonstrate sparse Transformer and MobileNet models that achieve 1.2-2.1x speedups and up to 12.8x memory savings without sacrificing accuracy.

Key facts. Cited by 138 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 26; 2022: 23; 2023: 33; 2024: 24; 2025: 18; 2026: 14.

138 papers.

Sort:

By area:

2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
2023Challenges and Applications of Large Language Modelscited by 251
2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 128
2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
2021The Efficiency Misnomercited by 98
2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61

38 more

2022SQuant: On-the-Fly Data-Free Quantization via Diagonal Hessian Approximationcited by 51
2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
2022SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learningcited by 47
2021The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translationcited by 46
2021Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsitycited by 44
2023Dynamic Sparse Training with Structured Sparsitycited by 36
2024AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Modelscited by 35
2022SPDY: Accurate Pruning with Speedup Guaranteescited by 34
2023SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Modelscited by 26
2023PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformationcited by 25
2023Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researcherscited by 25
2023Sparse Modular Activation for Efficient Sequence Modelingcited by 20
2021Transformer Acceleration with Dynamic Sparse Attentioncited by 19
2022On-device Training: A First Overview on Existing Systemscited by 18
2021The CoRa Tensor Compiler: Compilation for Ragged Tensors with Minimal Paddingcited by 17
2023Dynamic Sparsity Is Channel-Level Sparsity Learnercited by 17
2022Interspace Pruning: Using Adaptive Filter Representations to Improve Training of Sparse CNNscited by 16
2022Efficient Quantized Sparse Matrix Operations on Tensor Corescited by 16
2024Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuningcited by 16
2021Truly Sparse Neural Networks at Scalecited by 15
2021Two Sparsities Are Better Than One: Unlocking the Performance Benefits of Sparse-Sparse Networkscited by 15
2024LSK3DNet: Towards Effective and Efficient 3D Perception with Large Sparse Kernelscited by 15
2023Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsitycited by 14
2023Sparse Fine-tuning for Inference Acceleration of Large Language Modelscited by 14
2021SparseDNN: Fast Sparse Deep Learning Inference on CPUscited by 13
2023JaxPruner: A concise library for sparsity researchcited by 13
2023VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Corescited by 13
2024MULTIFLOW: Shifting Towards Task-Agnostic Vision-Language Pruningcited by 13
2021CogDL: A Comprehensive Library for Graph Deep Learningcited by 12
2022Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Surveycited by 12
2023SparseProp: Efficient Sparse Backpropagation for Faster Training of Neural Networkscited by 12
2024GNNShap: Scalable and Accurate GNN Explanation using Shapley Valuescited by 12
2022BaCO: A Fast and Portable Bayesian Compiler Optimization Frameworkcited by 11
2023Sparse Distributed Memory is a Continual Learnercited by 11
2023A2Q: Accumulator-Aware Quantization with Guaranteed Overflow Avoidancecited by 11
2021BlockCopy: High-Resolution Video Processing with Block-Sparse Feature Propagation and Online Policiescited by 9
2023Reparo: Loss-Resilient Generative Codec for Video Conferencingcited by 8
2025Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boostcited by 8

50 most cited of 138 All 138 (JSON)