mlsanityData to 7 Oct 2026

A Comprehensive Survey of Compression Algorithms for Language Models

arXiv 2401.15347 ↗First posted 27 Jan 2024cs.CL cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

How can we compress language models without sacrificing accuracy? The number of compression algorithms for language models is rapidly growing to benefit from remarkable advances of recent language models without side effects due to the gigantic size of language models, such as increased carbon emissions and expensive maintenance fees. While numerous compression algorithms have shown remarkable progress in compressing language models, it ironically becomes challenging to capture emerging trends and identify the fundamental concepts underlying them due to the excessive number of algorithms. In this paper, we survey and summarize diverse compression algorithms including pruning, quantization, knowledge distillation, low-rank approximation, parameter sharing, and efficient architecture design. We not only summarize the overall trend of diverse compression algorithms but also select representative algorithms and provide in-depth analyses of them. We discuss the value of each category of compression algorithms, and the desired properties of low-cost compression algorithms which have a significant impact due to the emergence of large language models. Finally, we introduce promising future research topics based on our survey results.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 126 of its 169 references that are in the map. First posted on arXiv 27 Jan 2024.

Builds on

126 of 169 references.

References · 126

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  11. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  12. 2016Layer Normalizationcited by 5,579
38 more
  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  13. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  14. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  15. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  16. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  17. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  18. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  19. 2020Reformer: The Efficient Transformercited by 1,658
  20. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  21. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  22. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  23. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  24. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  25. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  26. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  27. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  28. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  29. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  30. 2019Are Sixteen Heads Really Better than One?cited by 803
  31. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  32. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  33. 2018Universal Transformerscited by 613
  34. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  35. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  36. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  37. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  38. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476

50 most cited of 126 All 126 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 7; 2026: 2.

15 papers.

Sort:

By area:

  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  2. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  3. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 142
  4. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  5. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  6. 2025Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysiscited by 12
  7. 2024The Impact of Inference Acceleration on Bias of LLMscited by 9
  8. 2025Zero-shot Quantization: A Comprehensive Surveycited by 7
  9. 2025Unifying Uniform and Binary-coding Quantization for Accurate Compression of Large Language Modelscited by 5
  10. 2025Edge Deployment of Small Language Models, a comprehensive comparison of CPU, GPU and NPU backendscited by 4
  11. 2025Lossless Compression for LLM Tensor Incremental Snapshotscited by 2
  12. 2025Accurate Sublayer Pruning for Large Language Models by Exploiting Latency and Tunability Informationcited by 2
3 more
  1. 2026KD4MT: A Survey of Knowledge Distillation for Machine Translationcited by 2
  2. 2026SynQ: Accurate Zero-shot Quantization by Synthesis-aware Fine-tuningcited by 1
  3. 2025Striking the Right Balance between Compute and Copy: Improving LLM Inferencing Under Speculative Decodingcited by 0