mlsanityData to 7 Oct 2026

Evaluating Quantized Large Language Models

arXiv 2402.18158 ↗First posted 28 Feb 2024cs.CL cs.AIcited by 75

Not on the map. Why?

Abstract (arXiv)

Post-training quantization (PTQ) has emerged as a promising technique to reduce the cost of large language models (LLMs). Specifically, PTQ can effectively mitigate memory consumption and reduce computational overhead in LLMs. To meet the requirements of both high efficiency and performance across diverse scenarios, a comprehensive evaluation of quantized LLMs is essential to guide the selection of quantization methods. This paper presents a thorough evaluation of these factors by evaluating the effect of PTQ on Weight, Activation, and KV Cache on 11 model families, including OPT, LLaMA2, Falcon, Bloomz, Mistral, ChatGLM, Vicuna, LongChat, StableLM, Gemma, and Mamba, with parameters ranging from 125M to 180B. The evaluation encompasses five types of tasks: basic NLP, emergent ability, trustworthiness, dialogue, and long-context tasks. Moreover, we also evaluate the state-of-the-art (SOTA) quantization methods to demonstrate their applicability. Based on the extensive experiments, we systematically summarize the effect of quantization, provide recommendations to apply quantization techniques, and point out future directions. The code can be found in https://github.com/thu-nics/qllm-eval.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Builds on 43 of its 55 references that are in the map. First posted on arXiv 28 Feb 2024.

Builds on

43 of 55 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
31 more
  1. 2023Mistral 7Bcited by 3,065
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  4. 2022Emergent Abilities of Large Language Modelscited by 2,483
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  7. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  9. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  10. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  12. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  13. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  14. 2020Aligning AI With Shared Human Valuescited by 706
  15. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  16. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  17. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  18. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  19. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
  20. 2021A White Paper on Neural Network Quantizationcited by 453
  21. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  22. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  23. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  24. 2023Efficient Large Language Models: A Surveycited by 183
  25. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
  26. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
  27. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 112
  28. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 80
  29. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 78
  30. 2019ChID: A Large-scale Chinese IDiom Dataset for Cloze Testcited by 65
  31. 2023Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Studycited by 43

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 32; 2026: 20.

75 papers.

Sort:

By area:

  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  2. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  3. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 116
  4. 2025Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Modelscited by 68
  5. 2024Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decodingcited by 65
  6. 2024MBQ: Modality-Balanced Quantization for Large Vision-Language Modelscited by 45
  7. 2024Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengthscited by 44
  8. 2024Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Modelscited by 40
  9. 2025Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latencycited by 32
  10. 2025Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantizationcited by 30
  11. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 28
  12. 202570% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)cited by 23
38 more
  1. 2025Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMscited by 21
  2. 2024Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scalecited by 20
  3. 2024Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Mergingcited by 18
  4. 2024AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inferencecited by 17
  5. 2025Investigating the Impact of Quantization Methods on the Safety and Reliability of Large Language Modelscited by 17
  6. 2025PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Modelscited by 17
  7. 2025DarwinLM: Evolutionary Structured Pruning of Large Language Modelscited by 16
  8. 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 15
  9. 2024Mixture of Scales: Memory-Efficient Token-Adaptive Binarization for Large Language Modelscited by 14
  10. 2025Pre-Trained Policy Discriminators are General Reward Modelscited by 14
  11. 2024Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giantcited by 12
  12. 2025Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysiscited by 12
  13. 2024Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolboxcited by 11
  14. 2024Channel-Wise Mixed-Precision Quantization for Large Language Modelscited by 11
  15. 2025Inference-time sparse attention with asymmetric indexingcited by 11
  16. 2025LightMamba: Efficient Mamba Acceleration on FPGA with Quantization and Hardware Co-designcited by 11
  17. 2025Low-bit Model Quantization for Deep Neural Networks: A Surveycited by 11
  18. 2024KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Servingcited by 10
  19. 2024INT-FlashAttention: Enabling Flash Attention for INT8 Quantizationcited by 9
  20. 2026What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Studycited by 9
  21. 2026Pushing the Limits of Block Rotations in Post-Training Quantizationcited by 9
  22. 2026Pre-Registering the Detectable Effect: A Paired-MDE Budget for 4-bit Quantization Benchmarks, with a Pilot Auditcited by 8
  23. 2024Vision-Language and Large Language Model Performance in Gastroenterology: GPT, Claude, Llama, Phi, Mistral, Gemma, and Quantized Modelscited by 7
  24. 2025Pushing up to the Limit of Memory Bandwidth and Capacity Utilization for Efficient LLM Decoding on Embedded FPGAcited by 7
  25. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 7
  26. 2025Unifying Block-wise PTQ and Distillation-based QAT for Progressive Quantization toward 2-bit Instruction-Tuned LLMscited by 7
  27. 2025Edge-ASR: Towards Low-Bit Quantization of Automatic Speech Recognition Modelscited by 5
  28. 2024Accumulator-Aware Post-Training Quantization for Large Language Modelscited by 4
  29. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospectscited by 4
  30. 2025The Devil in the Details: Emergent Misalignment, Format and Coherence in Open-Weights LLMscited by 4
  31. 2026Perplexity Can Miss SAE Feature Damage Under Quantizationcited by 4
  32. 2024The Impact of Quantization on Retrieval-Augmented Generation: An Analysis of Small LLMscited by 3
  33. 2024Evaluating Quantized Large Language Models for Code Generation on Low-Resource Language Benchmarkscited by 3
  34. 2025Soup-of-Experts: Pretraining Specialist Models via Parameters Averagingcited by 3
  35. 2025RSQ: Learning from Important Tokens Leads to Better Quantized LLMscited by 3
  36. 2025Quantized Large Language Models in Biomedical Natural Language Processing: Evaluation and Recommendationcited by 3
  37. 2026Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantizationcited by 3
  38. 2025Distilling Calibration via Conformalized Credal Inferencecited by 2

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Post-Training Quantization: Introducing a Statistical Pre-Calibration Approach
  2. 2026SliderQuant: Accurate Post-Training Quantization for LLMs
  3. 2025Rethinking Output Alignment For 1-bit Post-Training Quantization of Large Language Models
  4. 2025PTQTP: Post-Training Quantization to Trit-Planes for Large Language Models
  5. 2026C-PTQ: Fisher-weighted Channel-wise Sensitivity for Post-training Quantization of MLLMs
  6. 2025LLM Compression: How Far Can We Go in Balancing Size and Performance?
  7. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More
  8. 2024ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models