mlsanityData to 14 Oct 2026

LLM-Pruner: On the Structural Pruning of Large Language Models

arXiv 2305.11627 ↗First posted 19 May 2023cs.CLcited by 650

Not on the map. Why?

Cited by 61 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have shown remarkable capabilities in language understanding and generation. However, such impressive capability typically comes with a substantial model size, which presents significant challenges in both the deployment, inference, and training stages. With LLM being a general-purpose task solver, we explore its compression in a task-agnostic manner, which aims to preserve the multi-task solving and language generation ability of the original LLM. One challenge to achieving this is the enormous size of the training corpus of LLM, which makes both data transfer and model post-training over-burdensome. Thus, we tackle the compression of LLMs within the bound of two constraints: being task-agnostic and minimizing the reliance on the original training dataset. Our method, named LLM-Pruner, adopts structural pruning that selectively removes non-critical coupled structures based on gradient information, maximally preserving the majority of the LLM's functionality. To this end, the performance of pruned models can be efficiently recovered through tuning techniques, LoRA, in merely 3 hours, requiring only 50K data. We validate the LLM-Pruner on three LLMs, including LLaMA, Vicuna, and ChatGLM, and demonstrate that the compressed models still exhibit satisfactory capabilities in zero-shot classification and generation. The code is available at: https://github.com/horseee/LLM-Pruner

Key facts. Cited by 650 papers in the map (data through 14 Oct 2026). Builds on 63 of its 75 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

63 of 75 references.

References · 63

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  12. 2016Pointer Sentinel Mixture Modelscited by 3,096
38 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  4. 2022Emergent Abilities of Large Language Modelscited by 2,512
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  7. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  12. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  13. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  14. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  15. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  16. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  17. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  18. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  19. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  20. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 652
  21. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  22. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  23. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  24. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  25. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
  26. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  27. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  28. 2019Structured Pruning of Large Language Modelscited by 203
  29. 2023DepGraph: Towards Any Structural Pruningcited by 203
  30. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 178
  31. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
  32. 2022A Fast Post-Training Pruning Framework for Transformerscited by 162
  33. 2020Distilling Knowledge from Graph Convolutional Networkscited by 129
  34. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  35. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  36. 2022Dataset Distillation via Factorizationcited by 115
  37. 2015Data-free parameter pruning for Deep Neural Networkscited by 112
  38. 2019EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasiscited by 98

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 47; 2024: 192; 2025: 244; 2026: 167.

650 papers.

Sort:

By area:

  1. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  2. 2023A Comprehensive Overview of Large Language Modelscited by 648
  3. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  4. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  5. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  6. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  7. 2023A Survey on Model Compression for Large Language Modelscited by 269
  8. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  9. 2023Challenges and Applications of Large Language Modelscited by 254
  10. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  11. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  12. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
38 more
  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  2. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 202
  3. 2023Efficient Large Language Models: A Surveycited by 186
  4. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 165
  5. 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
  6. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
  7. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  8. 2024Real-Time Video Generation with Pyramid Attention Broadcastcited by 149
  9. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
  10. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 147
  11. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 131
  12. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  13. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 122
  14. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 117
  15. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  16. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  17. 2024SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blockscited by 97
  18. 2023Diffusion Model as Representation Learnercited by 95
  19. 2025Efficient Reasoning Models: A Surveycited by 95
  20. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 94
  21. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  22. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 82
  23. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 79
  24. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 76
  25. 2024CLLMs: Consistency Large Language Modelscited by 72
  26. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  27. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 69
  28. 2023GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graphcited by 69
  29. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 68
  30. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Modelscited by 64
  31. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  32. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 62
  33. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 62
  34. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
  35. 2024Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compressioncited by 57
  36. 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
  37. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 55
  38. 2024Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMscited by 54

50 most cited of 650 All 650 (JSON)