mlsanityData to 14 Oct 2026

A Simple and Effective Pruning Approach for Large Language Models

arXiv 2306.11695 ↗First posted 20 Jun 2023cs.CL cs.AI cs.LGcited by 800

Not on the map. Why?

Cited by 63 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

As their size increases, Large Languages Models (LLMs) are natural candidates for network pruning methods: approaches that drop a subset of network weights while striving to preserve performance. Existing methods, however, require either retraining, which is rarely affordable for billion-scale LLMs, or solving a weight reconstruction problem reliant on second-order information, which may also be computationally expensive. In this paper, we introduce a novel, straightforward yet effective pruning method, termed Wanda (Pruning by Weights and activations), designed to induce sparsity in pretrained LLMs. Motivated by the recent observation of emergent large magnitude features in LLMs, our approach prunes weights with the smallest magnitudes multiplied by the corresponding input activations, on a per-output basis. Notably, Wanda requires no retraining or weight update, and the pruned LLM can be used as is. We conduct a thorough evaluation of our method Wanda on LLaMA and LLaMA-2 across various language benchmarks. Wanda significantly outperforms the established baseline of magnitude pruning and performs competitively against recent method involving intensive weight update. Code is available at https://github.com/locuslab/wanda.

Key facts. Cited by 800 papers in the map (data through 14 Oct 2026). Builds on 87 of its 108 references that are in the map. First posted on arXiv 20 Jun 2023.

Builds on

87 of 108 references.

References · 87

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  12. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
38 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  3. 2022A ConvNet for the 2020scited by 3,579
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  8. 2022Emergent Abilities of Large Language Modelscited by 2,512
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  13. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  14. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  15. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  16. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  17. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  18. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  19. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  20. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 676
  21. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  22. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  23. 2018Rethinking the Value of Network Pruningcited by 599
  24. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  25. 2020What is the State of Neural Network Pruning?cited by 583
  26. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  27. 2019Importance Estimation for Neural Network Pruningcited by 541
  28. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  29. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  30. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  31. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  32. 2019The State of Sparsity in Deep Neural Networkscited by 476
  33. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  34. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  35. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  36. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  37. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  38. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 36; 2024: 214; 2025: 289; 2026: 261.

800 papers.

Sort:

By area:

  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  2. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  3. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  4. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  5. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  6. 2023A Survey on Model Compression for Large Language Modelscited by 269
  7. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  8. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  9. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  10. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  11. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  12. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 202
38 more
  1. 2023Efficient Large Language Models: A Surveycited by 186
  2. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 165
  3. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
  4. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  5. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 131
  6. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 122
  7. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 117
  8. 2026Ministral 3cited by 114
  9. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  10. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  11. 2024SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blockscited by 97
  12. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 94
  13. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 91
  14. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  15. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 82
  16. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 79
  17. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 76
  18. 2025Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Modelscited by 73
  19. 2024CLLMs: Consistency Large Language Modelscited by 72
  20. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 72
  21. 2024On the Role of Attention Heads in Large Language Model Safetycited by 72
  22. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 70
  23. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  24. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 69
  25. 2024Real-Time Anomaly Detection and Reactive Planning with Large Language Modelscited by 68
  26. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 68
  27. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Modelscited by 64
  28. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  29. 2024SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Modelscited by 63
  30. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 62
  31. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 62
  32. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
  33. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 61
  34. 2025Unlocking Efficient Long-to-Short LLM Reasoning with Model Mergingcited by 58
  35. 2024Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compressioncited by 57
  36. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decompositioncited by 57
  37. 2024ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruningcited by 55
  38. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 55

50 most cited of 800 All 800 (JSON)