mlsanityData to 7 Oct 2026

A Survey on Model Compression for Large Language Models

arXiv 2308.07633 ↗First posted 15 Aug 2023cs.CL cs.AIcited by 263

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have transformed natural language processing tasks successfully. Yet, their large size and high computational needs pose challenges for practical use, especially in resource-limited settings. Model compression has emerged as a key research area to address these challenges. This paper presents a survey of model compression techniques for LLMs. We cover methods like quantization, pruning, and knowledge distillation, highlighting recent advancements. We also discuss benchmarking strategies and evaluation metrics crucial for assessing compressed LLMs. This survey offers valuable insights for researchers and practitioners, aiming to enhance efficiency and real-world applicability of LLMs while laying a foundation for future advancements.

Key facts. Cited by 263 papers in the map (data through 7 Oct 2026). Builds on 119 of its 202 references that are in the map. First posted on arXiv 15 Aug 2023.

Builds on

119 of 202 references.

References · 119

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  10. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  11. 2020Scaling Laws for Neural Language Modelscited by 6,411
  12. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
38 more
  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  11. 2023A Survey of Large Language Modelscited by 2,537
  12. 2022Emergent Abilities of Large Language Modelscited by 2,483
  13. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  14. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  15. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  16. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  17. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  18. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  19. 2019Language Models as Knowledge Bases?cited by 1,922
  20. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  21. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  22. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  23. 2020Knowledge Distillation: A Surveycited by 1,578
  24. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  25. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  26. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  27. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  28. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  29. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  30. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  31. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  32. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  33. 2019Relational Knowledge Distillationcited by 873
  34. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  35. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  36. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  37. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  38. 2016Sequence-Level Knowledge Distillationcited by 761

50 most cited of 119 All 119 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 20; 2024: 83; 2025: 113; 2026: 47.

263 papers.

Sort:

By area:

  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  2. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  3. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  4. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  5. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  6. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
  7. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  8. 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 196
  9. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 156
  10. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 154
  11. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  12. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 137
38 more
  1. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  2. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  4. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 122
  5. 2024Scaling Laws for Precisioncited by 92
  6. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  7. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Lengthcited by 83
  8. 2023PB-LLM: Partially Binarized Large Language Modelscited by 80
  9. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 80
  10. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 77
  11. 2024Catastrophic Failure of LLM Unlearning via Quantizationcited by 76
  12. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 71
  13. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
  14. 2024SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Modelscited by 61
  15. 2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 45
  16. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 41
  17. 2024Accurate LoRA-Finetuning Quantization of LLMs via Information Retentioncited by 40
  18. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains Morecited by 38
  19. 2023Urban Generative Intelligence (UGI): A Foundational Platform for Agents in Embodied City Environmentcited by 37
  20. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  21. 2024REEF: Representation Encoding Fingerprints for Large Language Modelscited by 34
  22. 2024Large Language Models for UAVs: Current State and Pathways to the Futurecited by 33
  23. 2026Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Levelcited by 29
  24. 2023LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compressioncited by 26
  25. 2024Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trendscited by 26
  26. 2024Quamba: A Post-Training Quantization Recipe for Selective State Space Modelscited by 25
  27. 2025Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Surveycited by 25
  28. 2023Green Edge AI: A Contemporary Surveycited by 24
  29. 2025Text Data Augmentation for Large Language Models: A Comprehensive Survey of Methods, Challenges, and Opportunitiescited by 21
  30. 2025Algebra Unveils Deep Learning -- An Invitation to Neuroalgebraic Geometrycited by 21
  31. 2024Faster and Lighter LLMs: A Survey on Current Challenges and Way Forwardcited by 20
  32. 2024Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scalecited by 20
  33. 2023Graph Meets LLMs: Towards Large Graph Modelscited by 19
  34. 2024Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Mergingcited by 18
  35. 2025TR-DQ: Time-Rotation Diffusion Quantizationcited by 18
  36. 2025VLA-RAIL: A Real-Time Asynchronous Inference Linker for VLA Models and Robotscited by 18
  37. 2024Agreement-Based Cascading for Efficient Inferencecited by 16
  38. 2024On Efficient Variants of Segment Anything Model: A Surveycited by 16

50 most cited of 263 All 263 (JSON)