Data to 7 Oct 2026

The State of Sparsity in Deep Neural Networks

arXiv 1902.09574 ↗First posted 25 Feb 2019cs.LG stat.MLcited by 471

Not on the map. Why?

Abstract (arXiv)

We rigorously evaluate three state-of-the-art techniques for inducing sparsity in deep neural networks on two large-scale learning tasks: Transformer trained on WMT 2014 English-to-German, and ResNet-50 trained on ImageNet. Across thousands of experiments, we demonstrate that complex techniques (Molchanov et al., 2017; Louizos et al., 2017b) shown to yield high compression rates on smaller datasets perform inconsistently, and that simple magnitude pruning approaches achieve comparable or better results. Additionally, we replicate the experiments performed by (Frankle & Carbin, 2018) and (Liu et al., 2018) at scale and show that unstructured sparse architectures learned through pruning cannot be trained from scratch to the same test set performance as a model trained with joint sparsification and optimization. Together, these results highlight the need for large-scale benchmarks in the field of model compression. We open-source our code, top performing model checkpoints, and results of all hyperparameter configurations to establish rigorous baselines for future work on compression and sparsification.

Key facts. Cited by 471 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 101; 2022: 89; 2023: 105; 2024: 77; 2025: 65; 2026: 34.

471 papers.

Sort:

By area:

2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259

38 more

2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
2023Speculative Decoding with Big Little Decodercited by 163
2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
2022A Fast Post-Training Pruning Framework for Transformerscited by 161
2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 153
2022A Review of Sparse Expert Models in Deep Learningcited by 147
2021A Unified Lottery Ticket Hypothesis for Graph Neural Networkscited by 118
2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 100
2021The Efficiency Misnomercited by 98
2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98
2021Sparse is Enough in Scaling Transformerscited by 86
2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 80
2021Top-KAST: Top-K Always Sparse Trainingcited by 79
2022Unified Visual Transformer Compressioncited by 79
2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
2022Efficient Methods for Natural Language Processing: A Surveycited by 78
2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
2021Only Train Once: A One-Shot Neural Network Training And Pruning Frameworkcited by 70
2021Token Pooling in Vision Transformerscited by 68
2021Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracycited by 62
2021A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustnesscited by 62
2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
2022The State of Sparse Training in Deep Reinforcement Learningcited by 60
2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 59
2021AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networkscited by 58
2022Advancing Model Pruning via Bi-level Optimizationcited by 55
2021M-FAC: Efficient Matrix-Free Approximations of Second-Order Informationcited by 54
2021Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Bettercited by 54
2024Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compressioncited by 54
2021Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?cited by 52
2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50

50 most cited of 471 All 471 (JSON)