mlsanityData to 7 Oct 2026

The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models

arXiv 2312.00960 ↗First posted 1 Dec 2023cs.CL cs.AI cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Compressing large language models (LLMs), often consisting of billions of parameters, provides faster inference, smaller memory footprints, and enables local deployment. Two standard compression techniques are pruning and quantization, with the former eliminating redundant connections in model layers and the latter representing model parameters with fewer bits. The key tradeoff is between the degree of compression and the impact on the quality of the compressed model. Existing research on LLM compression primarily focuses on performance in terms of general metrics like perplexity or downstream task accuracy. More fine-grained metrics, such as those measuring parametric knowledge, remain significantly underexplored. To help bridge this gap, we present a comprehensive analysis across multiple model families (ENCODER, ENCODER-DECODER, and DECODER) using the LAMA and LM-HARNESS benchmarks in order to systematically quantify the effect of commonly employed compression techniques on model performance. A particular focus is on tradeoffs involving parametric knowledge, with the goal of providing practitioners with practical insights to help make informed decisions on compression. We release our codebase1 to enable further research.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 40 of its 53 references that are in the map. First posted on arXiv 1 Dec 2023.

Builds on

40 of 53 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2019Language Models as Knowledge Bases?cited by 1,922
28 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  2. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  3. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  4. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  5. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  6. 2019Are Sixteen Heads Really Better than One?cited by 803
  7. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  8. 2021GPT Understands, Toocited by 715
  9. 2019Green AIcited by 620
  10. 2021Fast Model Editing at Scalecited by 491
  11. 2019The State of Sparsity in Deep Neural Networkscited by 471
  12. 2021A White Paper on Neural Network Quantizationcited by 453
  13. 2022Memory-Based Model Editing at Scalecited by 443
  14. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  15. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  16. 2023The False Promise of Imitating Proprietary LLMscited by 254
  17. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  18. 2019Assessing BERT's Syntactic Abilitiescited by 191
  19. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  20. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  21. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  22. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  23. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  24. 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
  25. 2016Compression of Neural Machine Translation Models via Pruningcited by 43
  26. 2021BERTnesia: Investigating the capture and forgetting of knowledge in BERTcited by 40
  27. 2018Balanced Sparsity for Efficient DNN Inference on GPUcited by 27
  28. 2018Assessing Composition in Sentence Vector Representationscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 2; 2026: 2.

9 papers.

  1. 2024Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compressioncited by 54
  2. 2024Accuracy is Not All You Needcited by 27
  3. 2024FrameQuant: Flexible Low-Bit Quantization for Transformerscited by 17
  4. 2024Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giantcited by 12
  5. 2026Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuningcited by 9
  6. 2024Composable Interventions for Language Modelscited by 4
  7. 2025Is Compression Really Linear with Code Intelligence?cited by 2
  8. 2025Through a Compressed Lens: Investigating The Impact of Quantization on Factual Knowledge Recallcited by 2
  9. 2026Linguistic properties and model scale in brain encoding: from small to compressed language modelscited by 0