mlsanityData to 14 Oct 2026

Structured Pruning of Large Language Models

arXiv 1910.04732 ↗First posted 10 Oct 2019cs.CL cs.LG stat.MLcited by 203

Not on the map. Why?

Abstract (arXiv)

Large language models have recently achieved state of the art performance across a wide variety of natural language tasks. Meanwhile, the size of these models and their latency have significantly increased, which makes their usage costly, and raises an interesting question: do language models need to be large? We study this question through the lens of model compression. We present a generic, structured pruning approach by parameterizing each weight matrix using its low-rank factorization, and adaptively removing rank-1 components during training. On language modeling tasks, our structured approach outperforms other unstructured and block-structured pruning baselines at various compression levels, while achieving significant speedups during both training and inference. We also demonstrate that our method can be applied to pruning adaptive word embeddings in large language models, and to pruning the BERT model on several downstream fine-tuning classification benchmarks.

Key facts. Cited by 203 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 22; 2022: 27; 2023: 56; 2024: 52; 2025: 33; 2026: 13.

203 papers.

Sort:

By area:

  1. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  2. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  3. 2023A Comprehensive Overview of Large Language Modelscited by 648
  4. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  5. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  6. 2023Challenges and Applications of Large Language Modelscited by 254
  7. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
  8. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  9. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 211
  10. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 202
  11. 2021Block Pruning For Faster Transformerscited by 177
  12. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
38 more
  1. 2022A Fast Post-Training Pruning Framework for Transformerscited by 162
  2. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  3. 2021Learned Token Pruning for Transformerscited by 139
  4. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 132
  5. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  6. 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patchingcited by 80
  7. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  8. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  9. 2023X-Pruner: eXplainable Pruning for Vision Transformerscited by 68
  10. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  11. 2024Finding Transformer Circuits with Edge Pruningcited by 60
  12. 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
  13. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  14. 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
  15. 2024MaskLLM: Learnable Semi-Structured Sparsity for Large Language Modelscited by 50
  16. 2023ZipLM: Inference-Aware Structured Pruning of Language Modelscited by 49
  17. 2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 48
  18. 2021Effective Sparsification of Neural Networks with Global Sparsity Constraintcited by 45
  19. 2022Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systemscited by 44
  20. 2024Everybody Prune Now: Structured Pruning of LLMs with only Forward Passescited by 43
  21. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
  22. 2021Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigmcited by 40
  23. 2021FLAT: An Optimized Dataflow for Mitigating Attention Bottleneckscited by 39
  24. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 36
  25. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 35
  26. 2021NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Frameworkcited by 34
  27. 2023Masked Structural Growth for 2x Faster Language Model Pre-trainingcited by 33
  28. 2023Open-Source LLMs for Text Annotation: A Practical Guide for Model Setting and Fine-Tuningcited by 32
  29. 2021AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Modelscited by 31
  30. 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 31
  31. 2024A Survey of Small Language Modelscited by 31
  32. 2021Efficient Neural Network Training via Forward and Backward Propagation Sparsificationcited by 30
  33. 2023What Matters In The Structured Pruning of Generative Language Models?cited by 30
  34. 2022Parameter-Efficient Sparsity for Large Language Models Fine-Tuningcited by 27
  35. 2023Circuit Breaking: Removing Model Behaviors with Targeted Ablationcited by 27
  36. 2023Gradient-Free Structured Pruning with Unlabeled Datacited by 26
  37. 2023HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformerscited by 25
  38. 2023Matrix Compression via Randomized Low Rank and Low Precision Factorizationcited by 25

50 most cited of 203 All 203 (JSON)