mlsanityData to 14 Oct 2026

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

arXiv 2301.00774 ↗First posted 2 Jan 2023cs.LGcited by 1,121

Not on the map. Why?

Cited by 104 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We show for the first time that large-scale generative pretrained transformer (GPT) family models can be pruned to at least 50% sparsity in one-shot, without any retraining, at minimal loss of accuracy. This is achieved via a new pruning method called SparseGPT, specifically designed to work efficiently and accurately on massive GPT-family models. We can execute SparseGPT on the largest available open-source models, OPT-175B and BLOOM-176B, in under 4.5 hours, and can reach 60% unstructured sparsity with negligible increase in perplexity: remarkably, more than 100 billion weights from these models can be ignored at inference time. SparseGPT generalizes to semi-structured (2:4 and 4:8) patterns, and is compatible with weight quantization approaches. The code is available at: https://github.com/IST-DASLab/sparsegpt.

Key facts. Cited by 1,121 papers in the map (data through 14 Oct 2026). Builds on 39 of its 56 references that are in the map. First posted on arXiv 2 Jan 2023.

Builds on

39 of 56 references.

References · 39

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  10. 2020Knowledge Distillation: A Surveycited by 1,603
  11. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  12. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
27 more
  1. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  2. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  3. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  4. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  5. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  6. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  7. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  8. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  9. 2019The State of Sparsity in Deep Neural Networkscited by 476
  10. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  11. 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 378
  12. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  13. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  14. 2021Accelerating Sparse Deep Neural Networkscited by 223
  15. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 179
  16. 2022A Fast Post-Training Pruning Framework for Transformerscited by 162
  17. 2021Group Fisher Pruning for Practical Network Compressioncited by 131
  18. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  19. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  20. 2019Fast Sparse ConvNetscited by 86
  21. 2021AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networkscited by 59
  22. 2018A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Datasetcited by 55
  23. 2021M-FAC: Efficient Matrix-Free Approximations of Second-Order Informationcited by 54
  24. 2021Playing Lottery Tickets with Vision and Languagecited by 45
  25. 2022SPDY: Accurate Pruning with Speedup Guaranteescited by 36
  26. 2022GMP*: Well-Tuned Gradual Magnitude Pruning Can Outperform Most BERT-Pruning Methodscited by 15
  27. 2022Quadapter: Adapter for GPT-2 Quantizationcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 102; 2024: 322; 2025: 371; 2026: 326.

1,121 papers.

Sort:

By area:

  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  2. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  3. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  4. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  5. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  6. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  7. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  8. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  9. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  10. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  11. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  12. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
38 more
  1. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  2. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
  3. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  4. 2023A Survey on Model Compression for Large Language Modelscited by 269
  5. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  6. 2023Challenges and Applications of Large Language Modelscited by 254
  7. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  8. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  9. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  10. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  11. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  12. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  13. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  14. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 202
  15. 2023Efficient Large Language Models: A Surveycited by 186
  16. 2023An LLM Compiler for Parallel Function Callingcited by 179
  17. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 174
  18. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  19. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  20. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 165
  21. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
  22. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  23. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  24. 2023REST: Retrieval-Based Speculative Decodingcited by 150
  25. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 147
  26. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
  27. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
  28. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  29. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  30. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  31. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 126
  32. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 122
  33. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 121
  34. 2024C3oT: Generating Shorter Chain-of-Thought without Compromising Effectivenesscited by 119
  35. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decodingcited by 117
  36. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 117
  37. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 116
  38. 2023Fluctuation-based Adaptive Structured Pruning for Large Language Modelscited by 111

50 most cited of 1,121 All 1,121 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
  2. 2026Debias-SparseGPT: Bias-Aware Pruning for Large Language Models
  3. 2021Prune Once for All: Sparse Pre-Trained Language Models
  4. 2022Sparse*BERT: Sparse Models Generalize To New tasks and Domains
  5. 2024Q-Sparse: All Large Language Models can be Fully Sparsely-Activated
  6. 2023SqueezeLLM: Dense-and-Sparse Quantization
  7. 2026Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity
  8. 2026SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference