mlsanityData to 7 Oct 2026

Accelerating Sparse Deep Neural Networks

arXiv 2104.08378 ↗First posted 16 Apr 2021cs.LG cs.AI cs.ARcited by 213

Not on the map. Why?

Abstract (arXiv)

As neural network model sizes have dramatically increased, so has the interest in various techniques to reduce their parameter counts and accelerate their execution. An active area of research in this field is sparsity - encouraging zero values in parameters that can then be discarded from storage or computations. While most research focuses on high levels of sparsity, there are challenges in universally maintaining model accuracy as well as achieving significant speedups over modern matrix-math hardware. To make sparsity adoption practical, the NVIDIA Ampere GPU architecture introduces sparsity support in its matrix-math units, Tensor Cores. We present the design and behavior of Sparse Tensor Cores, which exploit a 2:4 (50%) sparsity pattern that leads to twice the math throughput of dense matrix units. We also describe a simple workflow for training networks that both satisfy 2:4 sparsity pattern requirements and maintain accuracy, verifying it on a wide range of common tasks and model architectures. This workflow makes it easy to prepare accurate models for efficient deployment on Sparse Tensor Cores.

Key facts. Cited by 213 papers in the map (data through 7 Oct 2026). Builds on 50 of its 61 references that are in the map. First posted on arXiv 16 Apr 2021.

Builds on

50 of 61 references.

References · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  9. 2014Going Deeper with Convolutionscited by 4,231
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
38 more
  1. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  2. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  3. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  4. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  5. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  6. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  7. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  8. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  9. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  10. 2018Rethinking the Value of Network Pruningcited by 595
  11. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  12. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  13. 2019Importance Estimation for Neural Network Pruningcited by 531
  14. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  15. 2017Dilated Residual Networkscited by 293
  16. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  17. 2018Efficient Neural Audio Synthesiscited by 218
  18. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 183
  19. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  20. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  21. 2015Structured Pruning of Deep Convolutional Neural Networkscited by 148
  22. 2018Improving Semantic Segmentation via Video Propagation and Label Relaxationcited by 134
  23. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  24. 2019Fast Sparse ConvNetscited by 86
  25. 2017Exploring Sparsity in Recurrent Neural Networkscited by 75
  26. 2016Pruning Convolutional Neural Networks for Resource Efficient Inferencecited by 47
  27. 2017Block-Sparse Recurrent Neural Networkscited by 38
  28. 2019Stabilizing the Lottery Ticket Hypothesiscited by 36
  29. 2018Balanced Sparsity for Efficient DNN Inference on GPUcited by 27
  30. 2018Accelerator-Aware Pruning for Convolutional Neural Networkscited by 22
  31. 2020Filter Sketch for Network Pruningcited by 20
  32. 2019Beyond Human-Level Accuracy: Computational Challenges in Deep Learningcited by 19
  33. 2018Structured Pruning for Efficient ConvNets via Incremental Regularizationcited by 15
  34. 2018Stacked U-Nets: A No-Frills Approach to Natural Image Segmentationcited by 9
  35. 2016DSD: Dense-Sparse-Dense Training for Deep Neural Networkscited by 7
  36. 2020Campfire: Compressible, Regularization-Free, Structured Sparse Training for Hardware Acceleratorscited by 7
  37. 2018SQuantizer: Simultaneous Learning for Both Sparse and Low-precision Neural Networkscited by 3
  38. 2018Language Modeling at Scalecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 21; 2023: 50; 2024: 42; 2025: 57; 2026: 31.

213 papers.

Sort:

By area:

  1. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  2. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  3. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  4. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  5. 2023Challenges and Applications of Large Language Modelscited by 251
  6. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
  7. 2021Group Fisher Pruning for Practical Network Compressioncited by 130
  8. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  9. 2023Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generationcited by 103
  10. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  11. 2021Global Vision Transformer Pruning with Hessian-Aware Saliencycited by 69
  12. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
38 more
  1. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  2. 2024Dynamic Diffusion Transformercited by 60
  3. 2021AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networkscited by 58
  4. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 58
  5. 2023On the Impact of Calibration Data in Post-training Quantization and Pruningcited by 51
  6. 2021How Well Do Sparse Imagenet Models Transfer?cited by 50
  7. 2024MaskLLM: Learnable Semi-Structured Sparsity for Large Language Modelscited by 49
  8. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  9. 2023Pruning vs Quantization: Which is Better?cited by 45
  10. 2024Everybody Prune Now: Structured Pruning of LLMs with only Forward Passescited by 42
  11. 2023Scaling Laws for Sparsely-Connected Foundation Modelscited by 41
  12. 2024BitDelta: Your Fine-Tune May Only Be Worth One Bitcited by 38
  13. 2023A Survey on Efficient Training of Transformerscited by 37
  14. 2023Dynamic Sparse Training with Structured Sparsitycited by 36
  15. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  16. 2022SPDY: Accurate Pruning with Speedup Guaranteescited by 34
  17. 2024STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruningcited by 30
  18. 2022An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformerscited by 28
  19. 2021Dual-side Sparse Tensor Corecited by 27
  20. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  21. 2023STEP: Learning N:M Structured Sparsity Masks from Scratch with Preconditioncited by 25
  22. 2024DISP-LLM: Dimension-Independent Structural Pruning for Large Language Modelscited by 24
  23. 2023Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Casescited by 23
  24. 2023A Survey on Deep Learning Hardware Accelerators for Heterogeneous HPC Platformscited by 21
  25. 2022CAP: Correlation-Aware Pruning for Highly-Accurate Sparse Vision Modelscited by 20
  26. 2023Boost Vision Transformer with GPU-Friendly Sparsity and Quantizationcited by 20
  27. 2021RED++ : Data-Free Pruning of Deep Neural Networks via Input Splitting and Output Mergingcited by 19
  28. 2021NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMMcited by 19
  29. 2023ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Modelscited by 19
  30. 2024OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decompositioncited by 19
  31. 2025Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observationscited by 19
  32. 2022On-device Training: A First Overview on Existing Systemscited by 18
  33. 2024Accelerating Transformer Pre-training with 2:4 Sparsitycited by 18
  34. 2024SparseDM: Toward Sparse Efficient Diffusion Modelscited by 18
  35. 2021Post-Training Sparsity-Aware Quantizationcited by 16
  36. 20211xN Pattern for Pruning Convolutional Neural Networkscited by 16
  37. 2023Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarizationcited by 16
  38. 2024Sparse maximal update parameterization: A holistic approach to sparse training dynamicscited by 16

50 most cited of 213 All 213 (JSON)