# Can pruning make Large Language Models more efficient?

arXiv: https://arxiv.org/abs/2310.04573 · HTML: https://mlsanity.com/paper/2310.04573/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 6 Oct 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [Reducing Transformer Depth on Demand with Structured Dropout](https://mlsanity.com/paper/1909.11556/) (2019)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)
- [EIE: Efficient Inference Engine on Compressed Deep Neural Network](https://mlsanity.com/paper/1602.01528/) (2016)
- [Structured Pruning of Large Language Models](https://mlsanity.com/paper/1910.04732/) (2019)
- [Block Pruning For Faster Transformers](https://mlsanity.com/paper/2109.04838/) (2021)
- [On the Effect of Dropping Layers of Pre-trained Transformer Models](https://mlsanity.com/paper/2004.03844/) (2020)
- [Structured Bayesian Pruning via Log-Normal Multiplicative Noise](https://mlsanity.com/paper/1705.07283/) (2017)
- [Rocket Launching: A Universal and Efficient Framework for Training Well-performing Light Net](https://mlsanity.com/paper/1708.04106/) (2017)
- [Do Generative Large Language Models need billions of parameters?](https://mlsanity.com/paper/2309.06589/) (2023)
- [Zero-Shot Open-Book Question Answering](https://mlsanity.com/paper/2111.11520/) (2021)
- [Alexa, Predict My Flight Delay](https://mlsanity.com/paper/2208.09921/) (2022)
- [Can a student Large Language Model perform as well as it's teacher?](https://mlsanity.com/paper/2310.02421/) (2023)

## Built on by (most cited first, 5 of 5)

- [Model Compression and Efficient Inference for Large Language Models: A Survey](https://mlsanity.com/paper/2402.09748/) (2024)
- [Does Synthetic Data Make Large Language Models More Efficient?](https://mlsanity.com/paper/2310.07830/) (2023)
- [An exploration of the effect of quantisation on energy consumption and inference time of StarCoder2](https://mlsanity.com/paper/2411.12758/) (2024)
- [Fragile Knowledge, Robust Instruction-Following: The Width Pruning Dichotomy in Llama-3.2](https://mlsanity.com/paper/2512.22671/) (2025)
- [CascadedViT: Cascaded Chunk-FeedForward and Cascaded Group Attention Vision Transformer](https://mlsanity.com/paper/2511.14111/) (2025)

## Abstract (arXiv)

Transformer models have revolutionized natural language processing with their unparalleled ability to grasp complex contextual relationships. However, the vast number of parameters in these models has raised concerns regarding computational efficiency, environmental impact, and deployability on resource-limited platforms. To address these challenges, this paper investigates the application of weight pruning-a strategic reduction of model parameters based on their significance-as an optimization strategy for Transformer architectures. Through extensive experimentation, we explore various pruning methodologies, highlighting their impact on model performance, size, and computational demands. Our findings suggest that with judicious selection of pruning hyperparameters, significant reductions in model size are attainable without considerable compromise on performance. Moreover, when coupled with post-pruning fine-tuning strategies, some pruned models even exhibit enhanced generalization capabilities. This work seeks to bridge the gap between model efficiency and performance, paving the way for more scalable and environmentally responsible deep learning applications.