mlsanityData to 7 Oct 2026

Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks

arXiv 2102.00554 ↗First posted 31 Jan 2021cs.LG cs.AI cs.ARcited by 521

Not on the map. Why?

Abstract (arXiv)

The growing energy and performance costs of deep learning have driven the community to reduce the size of neural networks by selectively pruning components. Similarly to their biological counterparts, sparse networks generalize just as well, if not better than, the original dense networks. Sparsity can reduce the memory footprint of regular networks to fit mobile devices, as well as shorten training time for ever growing networks. In this paper, we survey prior work on sparsity in deep learning and provide an extensive tutorial of sparsification for both inference and training. We describe approaches to remove and add elements of neural networks, different training strategies to achieve model sparsity, and mechanisms to exploit sparsity in practice. Our work distills ideas from more than 300 research papers and provides guidance to practitioners who wish to utilize sparsity today, as well as to researchers whose goal is to push the frontier forward. We include the necessary background on mathematical methods in sparsification, describe phenomena such as early structure adaptation, the intricate relations between sparsity and the training process, and show techniques for achieving acceleration on real hardware. We also define a metric of pruned parameter efficiency that could serve as a baseline for comparison of different sparse networks. We close by speculating on how sparsity can improve future workloads and outline major open problems in the field.

Key facts. Cited by 521 papers in the map (data through 7 Oct 2026). Builds on 6 of its 425 references that are in the map. First posted on arXiv 31 Jan 2021.

Builds on

6 of 425 references.

References · 6

  1. 2014Going Deeper with Convolutionscited by 4,231
  2. 2017Decoupled Weight Decay Regularizationcited by 1,722
  3. 2018Rethinking the Value of Network Pruningcited by 595
  4. 2017WRPN: Wide Reduced-Precision Networkscited by 49
  5. 2018Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Boundscited by 43
  6. 2014Memory Bounded Deep Convolutional Networkscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 36; 2022: 85; 2023: 121; 2024: 124; 2025: 99; 2026: 56.

521 papers.

Sort:

By area:

  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  2. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  3. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  4. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  5. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  6. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  8. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  9. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  10. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  11. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 280
  12. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
38 more
  1. 2022Extremely Simple Activation Shaping for Out-of-Distribution Detectioncited by 212
  2. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  3. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 142
  4. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  5. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  6. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 120
  7. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  8. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 95
  9. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 94
  10. 2022Robust Training under Label Noise by Over-parameterizationcited by 93
  11. 2024AI and Memory Wallcited by 89
  12. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
  13. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  14. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
  15. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  16. 2022Deep Differentiable Logic Gate Networkscited by 67
  17. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  18. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 60
  19. 2021AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networkscited by 58
  20. 2022Advancing Model Pruning via Bi-level Optimizationcited by 55
  21. 2023On the Impact of Calibration Data in Post-training Quantization and Pruningcited by 51
  22. 2021How Well Do Sparse Imagenet Models Transfer?cited by 50
  23. 2024Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmeticcited by 50
  24. 2021Dynamic Sparse Training for Deep Reinforcement Learningcited by 49
  25. 2021Meta-Learning Sparse Implicit Neural Representationscited by 48
  26. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  27. 2022SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learningcited by 47
  28. 2023ZipLM: Inference-Aware Structured Pruning of Language Modelscited by 46
  29. 2024Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitatecited by 46
  30. 2024Kolmogorov-Arnold Network for Satellite Image Classification in Remote Sensingcited by 46
  31. 2022ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsitycited by 45
  32. 2021Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsitycited by 44
  33. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 44
  34. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 43
  35. 2024SlimGPT: Layer-wise Structured Pruning for Large Language Modelscited by 43
  36. 2023QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelscited by 42
  37. 2021Recent Advances on Neural Network Pruning at Initializationcited by 41
  38. 2023Scaling Laws for Sparsely-Connected Foundation Modelscited by 41

50 most cited of 521 All 521 (JSON)