# What Matters In The Structured Pruning of Generative Language Models?

arXiv: https://arxiv.org/abs/2302.03773 · HTML: https://mlsanity.com/paper/2302.03773/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 56 references that are in the map.
- First posted on arXiv 7 Feb 2023.

## Builds on (most cited first, 30 of 46)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization](https://mlsanity.com/paper/1911.12237/) (2019)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression](https://mlsanity.com/paper/1707.06342/) (2017)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)
- [AMC: AutoML for Model Compression and Acceleration on Mobile Devices](https://mlsanity.com/paper/1802.03494/) (2018)
- [Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning](https://mlsanity.com/paper/2012.09816/) (2020)
- [SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning](https://mlsanity.com/paper/2012.09852/) (2020)
- [NISP: Pruning Networks using Neuron Importance Score Propagation](https://mlsanity.com/paper/1711.05908/) (2017)
- [Structured Pruning of Large Language Models](https://mlsanity.com/paper/1910.04732/) (2019)
- [Block Pruning For Faster Transformers](https://mlsanity.com/paper/2109.04838/) (2021)
- [Structured Pruning Learns Compact and Accurate Models](https://mlsanity.com/paper/2204.00408/) (2022)
- [A Fast Post-Training Pruning Framework for Transformers](https://mlsanity.com/paper/2204.09656/) (2022)
- [Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning](https://mlsanity.com/paper/1611.05128/) (2016)

## Built on by (most cited first, 30 of 30)

- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023)
- [A Survey on Model Compression for Large Language Models](https://mlsanity.com/paper/2308.07633/) (2023)
- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems](https://mlsanity.com/paper/2312.15234/) (2023)
- [ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models](https://mlsanity.com/paper/2310.04564/) (2023)
- [A Survey of Resource-efficient LLM and Multimodal Foundation Models](https://mlsanity.com/paper/2401.08092/) (2024)
- [Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods](https://mlsanity.com/paper/2402.02834/) (2024)
- [Model Compression and Efficient Inference for Large Language Models: A Survey](https://mlsanity.com/paper/2402.09748/) (2024)
- [A Survey on Transformer Compression](https://mlsanity.com/paper/2402.05964/) (2024)
- [A Survey of Small Language Models](https://mlsanity.com/paper/2410.20011/) (2024)
- [Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generation](https://mlsanity.com/paper/2404.01365/) (2024)
- [Sliding-Window Merging for Compacting Patch-Redundant Layers in LLMs](https://mlsanity.com/paper/2502.19159/) (2025)
- [The Cost of Down-Scaling Language Models: Fact Recall Deteriorates before In-Context Learning](https://mlsanity.com/paper/2310.04680/) (2023)
- [NASH: A Simple Unified Framework of Structured Pruning for Accelerating Encoder-Decoder Language Models](https://mlsanity.com/paper/2310.10054/) (2023)
- [RankAdaptor: Hierarchical Rank Allocation for Efficient Fine-Tuning Pruned LLMs via Performance Model](https://mlsanity.com/paper/2406.15734/) (2024)
- [On Accelerating Edge AI: Optimizing Resource-Constrained Environments](https://mlsanity.com/paper/2501.15014/) (2025)
- [CPTQuant - A Novel Mixed Precision Post-Training Quantization Techniques for Large Language Models](https://mlsanity.com/paper/2412.03599/) (2024)
- [Exploring Activation Patterns of Parameters in Language Models](https://mlsanity.com/paper/2405.17799/) (2024)
- [Data Pruning in Generative Diffusion Models](https://mlsanity.com/paper/2411.12523/) (2024)
- [Extracting Interpretable Task-Specific Circuits from Large Language Models for Faster Inference](https://mlsanity.com/paper/2412.15750/) (2024)
- [When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language Models](https://mlsanity.com/paper/2502.15443/) (2025)
- [Efficient LLM inference solution on Intel GPU](https://mlsanity.com/paper/2401.05391/) (2023)
- [Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerations](https://mlsanity.com/paper/2408.03130/) (2024)
- [QPruner: Probabilistic Decision Quantization for Structured Pruning in Large Language Models](https://mlsanity.com/paper/2412.11629/) (2024)
- [ActVAR: Activating Mixtures of Weights and Tokens for Efficient Visual Autoregressive Generation](https://mlsanity.com/paper/2511.12893/) (2025)
- [Mixed Sparsity Training: Achieving 4× FLOP Reduction for Transformer Pretraining](https://mlsanity.com/paper/2408.11746/) (2024)
- [Cross-Domain Content Generation with Domain-Specific Small Language Models](https://mlsanity.com/paper/2409.17171/) (2024)
- [AutoMixQ: Self-Adjusting Quantization for High Performance Memory-Efficient Fine-Tuning](https://mlsanity.com/paper/2411.13814/) (2024)
- [SwiftPrune: Hessian-Free Weight Pruning for Large Language Models](https://mlsanity.com/paper/2501.16376/) (2025)
- [Post-Training Neural Network Pruning using Graph Curvature](https://mlsanity.com/paper/2601.16366/) (2026)

## Abstract (arXiv)

Auto-regressive large language models such as GPT-3 require enormous computational resources to use. Traditionally, structured pruning methods are employed to reduce resource usage. However, their application to and efficacy for generative language models is heavily under-explored. In this paper we conduct an comprehensive evaluation of common structured pruning methods, including magnitude, random, and movement pruning on the feed-forward layers in GPT-type models. Unexpectedly, random pruning results in performance that is comparable to the best established methods, across multiple natural language generation tasks. To understand these results, we provide a framework for measuring neuron-level redundancy of models pruned by different methods, and discover that established structured pruning methods do not take into account the distinctiveness of neurons, leaving behind excess redundancies. In view of this, we introduce Globally Unique Movement (GUM) to improve the uniqueness of neurons in pruned models. We then discuss the effects of our techniques on different redundancy metrics to explain the improved performance.