mlsanityData to 14 Oct 2026

A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations

arXiv 2308.06767 ↗First posted 13 Aug 2023cs.LG cs.CVcited by 202

Not on the map. Why?

Abstract (arXiv)

Modern deep neural networks, particularly recent large language models, come with massive model sizes that require significant computational and storage resources. To enable the deployment of modern models on resource-constrained environments and accelerate inference time, researchers have increasingly explored pruning techniques as a popular research direction in neural network compression. However, there is a dearth of up-to-date comprehensive review papers on pruning. To address this issue, in this survey, we provide a comprehensive review of existing research works on deep neural network pruning in a taxonomy of 1) universal/specific speedup, 2) when to prune, 3) how to prune, and 4) fusion of pruning and other compression techniques. We then provide a thorough comparative analysis of eight pairs of contrast settings for pruning and explore emerging topics, including pruning for large language models, large multimodal models, post-training pruning, and different supervision levels for pruning to shed light on the commonalities and differences of existing methods and lay the foundation for further method development. To facilitate future research, we build a curated collection of datasets, networks, and evaluations on different applications. Finally, we provide valuable recommendations on selecting pruning methods and prospect several promising research directions. We build a repository at https://github.com/hrcheng1066/awesome-pruning.

Key facts. Cited by 202 papers in the map (data through 14 Oct 2026). Builds on 244 of its 354 references that are in the map. First posted on arXiv 13 Aug 2023.

Builds on

244 of 354 references.

References · 244

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  7. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  8. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  9. 2023GPT-4 Technical Reportcited by 18,396
  10. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  11. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  12. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
38 more
  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  12. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  13. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  14. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  15. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  16. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  17. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  18. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  19. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  20. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  21. 2015SSD: Single Shot MultiBox Detectorcited by 3,130
  22. 2016Pointer Sentinel Mixture Modelscited by 3,096
  23. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  24. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  25. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  26. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  27. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  28. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  29. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  30. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  31. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  32. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  33. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  34. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,576
  35. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  36. 2016Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoisingcited by 1,358
  37. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,355
  38. 2016Pruning Filters for Efficient ConvNetscited by 1,143

50 most cited of 244 All 244 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 35; 2025: 105; 2026: 60.

202 papers.

Sort:

By area:

  1. 2024Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmeticcited by 50
  2. 2024A Survey on Transformer Compressioncited by 46
  3. 2025Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantizationcited by 41
  4. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
  5. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  6. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 20
  7. 2024Efficient Fine-Tuning and Concept Suppression for Pruned Diffusion Modelscited by 16
  8. 2024Training Compute Thresholds: Features and Functions in AI Regulationcited by 14
  9. 2025Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetriescited by 14
  10. 2024Not All Prompts Are Made Equal: Prompt-based Pruning of Text-to-Image Diffusion Modelscited by 13
  11. 2024SAFT: Towards Out-of-Distribution Generalization in Fine-Tuningcited by 12
  12. 2024Mixture of Efficient Diffusion Experts Through Automatic Interval and Sub-Network Selectioncited by 12
38 more
  1. 2025Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysiscited by 12
  2. 2025Astraea: A Token-wise Acceleration Framework for Video Diffusion Transformerscited by 11
  3. 2024Adversarial Pruning: A Survey and Benchmark of Pruning Methods for Adversarial Robustnesscited by 10
  4. 2024Layer-Adaptive State Pruning for Deep State Space Modelscited by 10
  5. 2025From Screens to Scenes: A Survey of Embodied AI in Healthcarecited by 10
  6. 2025FIT-Print: Towards False-claim-resistant Model Ownership Verification via Targeted Fingerprintcited by 10
  7. 2025Gatekeeper: Improving Model Cascades Through Confidence Tuningcited by 10
  8. 2025Continual Unlearning for Text-to-Image Diffusion Models: A Regularization Perspectivecited by 10
  9. 2025End-to-end workflow for machine learning-based qubit readout with QICK and hls4mlcited by 8
  10. 2025Joint Localization and Activation Editing for Low-Resource Fine-Tuningcited by 8
  11. 2026PreciseCache: Precise Feature Caching for Efficient and High-fidelity Video Generationcited by 8
  12. 2025Efficient and Robust Speaker Diarization via Structured Pruning of Self-Supervised Modelscited by 7
  13. 2025The Right to be Forgotten in Pruning: Unveil Machine Unlearning on Sparse Modelscited by 7
  14. 2025EA-ViT: Efficient Adaptation for Elastic Vision Transformercited by 7
  15. 2026Flash-VAED: Plug-and-Play VAE Decoders for Efficient Video Generationcited by 7
  16. 2025Adapt-Pruner: Adaptive Structural Pruning for Efficient Small Language Model Trainingcited by 6
  17. 2025NIRVANA: Structured Pruning Reimagined for Large Language Model Compressioncited by 6
  18. 2025Masking Teacher and Reinforcing Student for Distilling Vision-Language Modelscited by 6
  19. 2026Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularitiescited by 6
  20. 2024MINI-LLM: Memory-Efficient Structured Pruning for Large Language Modelscited by 5
  21. 2024Semantic Edge Computing and Semantic Communications in 6G Networks: A Unifying Survey and Research Challengescited by 5
  22. 2025A Survey on Dynamic Neural Networks: from Computer Vision to Multi-modal Sensor Fusioncited by 5
  23. 2025Efficient LLMs with AMP: Attention Heads and MLP Pruningcited by 5
  24. 2025Online Learning of Neural Networkscited by 5
  25. 2025An Analysis of Concept Bottleneck Models: Measuring, Understanding, and Mitigating the Impact of Noisy Annotationscited by 5
  26. 2024Redundant Queries in DETR-Based 3D Detection Methods: Unnecessary and Prunablecited by 4
  27. 2025Bridging Language Models and Financial Analysiscited by 4
  28. 2025Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMscited by 4
  29. 2025Scalable Interconnect Learning in Boolean Networkscited by 4
  30. 2025Understanding and Harnessing Sparsity in Unified Multimodal Modelscited by 4
  31. 2024Exploring Selective Layer Fine-Tuning in Federated Learningcited by 3
  32. 2024Towards Scalable Insect Monitoring: Ultra-Lightweight CNNs as On-Device Triggers for Insect Camera Trapscited by 3
  33. 2024Pruning Unrolled Networks (PUN) at Initialization for MRI Reconstruction Improves Generalizationcited by 3
  34. 2025Flow to Learn: Flow Matching on Neural Network Parameterscited by 3
  35. 2025Enhanced Pruning Strategy for Multi-Component Neural Architectures Using Component-Aware Graph Analysiscited by 3
  36. 2025Hardware/Software Co-Design of RISC-V Extensions for Accelerating Sparse DNNs on FPGAscited by 3
  37. 2025DYNAMAX: Dynamic computing for Transformers and Mamba based architecturescited by 3
  38. 2025Pruning by Block Benefit: Exploring the Properties of Vision Transformer Blocks during Domain Adaptationcited by 3

50 most cited of 202 All 202 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021Deep Neural Compression Via Concurrent Pruning and Self-Distillation
  2. 2025Pruning Laws for Large Language Models
  3. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Survey
  4. 2025Integrating Pruning with Quantization for Efficient Deep Neural Networks Compression
  5. 2019Structural Pruning in Deep Neural Networks: A Small-World Approach
  6. 2026Hybrid Compression: Integrating Pruning and Quantization for Optimized Neural Networks
  7. 2023How to Prune Your Language Model: Recovering Accuracy on the "Sparsity May Cry'' Benchmark
  8. 2024LD-Pruner: Efficient Pruning of Latent Diffusion Models using Task-Agnostic Insights