mlsanityData to 7 Oct 2026

What Matters In The Structured Pruning of Generative Language Models?

arXiv 2302.03773 ↗First posted 7 Feb 2023cs.CL cs.LGcited by 30

Not on the map. Why?

Abstract (arXiv)

Auto-regressive large language models such as GPT-3 require enormous computational resources to use. Traditionally, structured pruning methods are employed to reduce resource usage. However, their application to and efficacy for generative language models is heavily under-explored. In this paper we conduct an comprehensive evaluation of common structured pruning methods, including magnitude, random, and movement pruning on the feed-forward layers in GPT-type models. Unexpectedly, random pruning results in performance that is comparable to the best established methods, across multiple natural language generation tasks. To understand these results, we provide a framework for measuring neuron-level redundancy of models pruned by different methods, and discover that established structured pruning methods do not take into account the distinctiveness of neurons, leaving behind excess redundancies. In view of this, we introduce Globally Unique Movement (GUM) to improve the uniqueness of neurons in pruned models. We then discuss the effects of our techniques on different redundancy metrics to explain the improved performance.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 46 of its 56 references that are in the map. First posted on arXiv 7 Feb 2023.

Builds on

46 of 56 references.

References · 46

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  9. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  12. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
34 more
  1. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  2. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  3. 2018Rethinking the Value of Network Pruningcited by 595
  4. 2020What is the State of Neural Network Pruning?cited by 578
  5. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  6. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  7. 2019The State of Sparsity in Deep Neural Networkscited by 471
  8. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
  9. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  10. 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 375
  11. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  12. 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
  13. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  14. 2019Structured Pruning of Large Language Modelscited by 201
  15. 2021Block Pruning For Faster Transformerscited by 176
  16. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  17. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  18. 2016Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruningcited by 133
  19. 2015Data-free parameter pruning for Deep Neural Networkscited by 112
  20. 2019Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challengecited by 86
  21. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  22. 2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
  23. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  24. 2021Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarizationcited by 46
  25. 2021Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigmcited by 40
  26. 2020Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruningcited by 39
  27. 2021When to Prune? A Policy towards Early Structural Pruningcited by 38
  28. 2021Kronecker Decomposition for GPT Compressioncited by 27
  29. 2022Parameter-Efficient Sparsity for Large Language Models Fine-Tuningcited by 27
  30. 2022Can Model Compression Improve NLP Fairnesscited by 18
  31. 2020Learned Threshold Pruningcited by 16
  32. 2021A Short Study on Compressing Decoder-Based Language Modelscited by 16
  33. 2021DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Modelscited by 8
  34. 2021LEAP: Learnable Pruning for Transformer-based Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 16; 2025: 5; 2026: 1.

30 papers.

Sort:

By area:

  1. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  2. 2023A Survey on Model Compression for Large Language Modelscited by 263
  3. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  4. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  5. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  6. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  7. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 80
  8. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  9. 2024A Survey on Transformer Compressioncited by 44
  10. 2024A Survey of Small Language Modelscited by 31
  11. 2024Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generationcited by 20
  12. 2025Sliding-Window Merging for Compacting Patch-Redundant Layers in LLMscited by 8
18 more
  1. 2023NASH: A Simple Unified Framework of Structured Pruning for Accelerating Encoder-Decoder Language Modelscited by 7
  2. 2024RankAdaptor: Hierarchical Rank Allocation for Efficient Fine-Tuning Pruned LLMs via Performance Modelcited by 7
  3. 2023The Cost of Down-Scaling Language Models: Fact Recall Deteriorates before In-Context Learningcited by 6
  4. 2025On Accelerating Edge AI: Optimizing Resource-Constrained Environmentscited by 5
  5. 2024CPTQuant - A Novel Mixed Precision Post-Training Quantization Techniques for Large Language Modelscited by 4
  6. 2024Data Pruning in Generative Diffusion Modelscited by 3
  7. 2024Extracting Interpretable Task-Specific Circuits from Large Language Models for Faster Inferencecited by 3
  8. 2025When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language Modelscited by 3
  9. 2023Efficient LLM inference solution on Intel GPUcited by 2
  10. 2024Exploring Activation Patterns of Parameters in Language Modelscited by 2
  11. 2024Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerationscited by 2
  12. 2024QPruner: Probabilistic Decision Quantization for Structured Pruning in Large Language Modelscited by 2
  13. 2025ActVAR: Activating Mixtures of Weights and Tokens for Efficient Visual Autoregressive Generationcited by 1
  14. 2024Mixed Sparsity Training: Achieving 4× FLOP Reduction for Transformer Pretrainingcited by 0
  15. 2024Cross-Domain Content Generation with Domain-Specific Small Language Modelscited by 0
  16. 2024AutoMixQ: Self-Adjusting Quantization for High Performance Memory-Efficient Fine-Tuningcited by 0
  17. 2025SwiftPrune: Hessian-Free Weight Pruning for Large Language Modelscited by 0
  18. 2026Post-Training Neural Network Pruning using Graph Curvaturecited by 0