mlsanityData to 14 Oct 2026

LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit

arXiv 2405.06001 ↗First posted 9 May 2024cs.LG cs.AI cs.CLcited by 33

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Recent advancements in large language models (LLMs) are propelling us toward artificial general intelligence with their remarkable emergent abilities and reasoning capabilities. However, the substantial computational and memory requirements limit the widespread adoption. Quantization, a key compression technique, can effectively mitigate these demands by compressing and accelerating LLMs, albeit with potential risks to accuracy. Numerous studies have aimed to minimize the accuracy loss associated with quantization. However, their quantization configurations vary from each other and cannot be fairly compared. In this paper, we present LLMC, a plug-and-play compression toolkit, to fairly and systematically explore the impact of quantization. LLMC integrates dozens of algorithms, models, and hardwares, offering high extensibility from integer to floating-point quantization, from LLM to vision-language (VLM) model, from fixed-bit to mixed precision, and from quantization to sparsification. Powered by this versatile toolkit, our benchmark covers three key aspects: calibration data, algorithms (three strategies), and data formats, providing novel insights and detailed analyses for further research and practical guidance for users. Our toolkit is available at https://github.com/ModelTC/llmc.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 40 of its 59 references that are in the map. First posted on arXiv 9 May 2024.

Builds on

40 of 59 references.

References · 40

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2021Program Synthesis with Large Language Modelscited by 3,637
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
28 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  2. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  6. 2019Towards VQA Models That Can Readcited by 1,718
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  8. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  9. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  10. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  11. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  12. 2019Learned Step Size Quantizationcited by 628
  13. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  14. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  15. 2021A White Paper on Neural Network Quantizationcited by 472
  16. 2023Detecting Language Model Attacks with Perplexitycited by 431
  17. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  18. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300
  19. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 269
  20. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  21. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 238
  22. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  23. 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 189
  24. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 180
  25. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
  26. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 84
  27. 2021Diversifying Sample Generation for Accurate Data-Free Quantizationcited by 63
  28. 2021A Simple Approach to Jointly Rank Passages and Select Relevant Sentences in the OBQA Contextcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 16; 2026: 11.

33 papers.

Sort:

By area:

  1. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  2. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantizationcited by 34
  3. 2025Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantizationcited by 32
  4. 202570% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)cited by 24
  5. 2025Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Modelscited by 16
  6. 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 16
  7. 2024Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giantcited by 14
  8. 2025AMQ: Enabling AutoML for Mixed-precision Weight-Only Quantization of Large Language Modelscited by 14
  9. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruningcited by 13
  10. 2025MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skippingcited by 11
  11. 2025Does quantization affect models' performance on long-context tasks?cited by 10
  12. 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Formatcited by 9
21 more
  1. 2025Why Do Some Inputs Break Low-Bit LLM Quantization?cited by 9
  2. 2025SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Modelscited by 8
  3. 2026Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instructcited by 8
  4. 2026Pre-Registering the Detectable Effect: A Paired-MDE Budget for 4-bit Quantization Benchmarks, with a Pilot Auditcited by 8
  5. 2026Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusingcited by 7
  6. 2024Temporal Feature Matters: A Framework for Diffusion Model Quantizationcited by 5
  7. 2024PTSBench: A Comprehensive Post-Training Sparsity Benchmark Towards Algorithms and Modelscited by 5
  8. 2025Edge-ASR: Towards Low-Bit Quantization of Automatic Speech Recognition Modelscited by 5
  9. 2025How Quantization Shapes Bias in Large Language Modelscited by 4
  10. 2026The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMscited by 4
  11. 2025Beacon: Post-Training Quantization with Integrated Grid Selectioncited by 3
  12. 2026Token-level Data Selection for Safe LLM Fine-tuningcited by 3
  13. 2025SLMQuant:Benchmarking Small Language Model Quantization for Practical Deploymentcited by 2
  14. 2026Collaborative Few-Step Distillation and Low-Bit Quantization for Wan2.2 Dual-Expert Video Diffusion Modelscited by 2
  15. 2025BlindSight: Harnessing Sparsity for Efficient Vision-Language Modelscited by 0
  16. 2025Efficient Edge LLMs Deployment via HessianAware Quantization and CPU GPU Collaborativecited by 0
  17. 2026K-Quantization and its Impact on Output Performancecited by 0
  18. 2026The Joint Effect of Quantization and Sampling Temperature on LLM Safety Alignment: A Factorial Analysiscited by 0
  19. 2026Bits Under ZK-LLM: Evaluating Zero-Knowledge-Friendly Quantization for Verifiable Private LLM Inferencecited by 0
  20. 2026Replay the Curvature: Accurate and Scalable NVFP4 Quantization for Large Language Model Inferencecited by 0
  21. 2026Investigating Model Compression for Neural Machine Translation in the Biomedical Domaincited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox
  2. 2024A Comprehensive Evaluation of Quantization Strategies for Large Language Models
  3. 2023A Survey on Model Compression for Large Language Models
  4. 2024A Comprehensive Study on Quantization Techniques for Large Language Models
  5. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMs
  6. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms
  7. 2025Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysis
  8. 2024Foundations of Large Language Model Compression -- Part 1: Weight Quantization