mlsanityData to 14 Oct 2026

K-Quantization and its Impact on Output Performance

arXiv 2605.19645 ↗First posted 19 May 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent advancements in large language models (LLMs) have shown their remarkable capacities in many NLP tasks. However, their substantial size often presents challenges for deployment. This necessitates efficient techniques for model compression, with quantization emerging as a prominent solution. Despite its benefits, the exact impact of quantization (from 2- to 6-bit) on the performance and accuracy of LLMs remains an active area of research. This paper investigates the performance of eight LLMs at various quantization levels, focusing on tasks such as MMLU-Pro for knowledge processing and reasoning, CRUXEval for code comprehension, and MuSR for reading comprehension. Our results show a consistent trend where higher precision (e.g., 8-bit Q80) yields improved performance, albeit with diminishing returns. Aggressive quantization (e.g., 2-bit Q2K) usually retains acceptable accuracy, though some models show a substantial loss in performance. Our findings indicate that while lower bit precision generally reduces performance, the impact varies across models and tasks. Larger models show greater resilience to aggressive quantization, but can still undergo significant drops at lower precision levels. Mid-sized models in the 7-9 billion parameter range strike an optimal balance between efficiency and resource usage. Such results provide insights into the trade-offs between model size, quantization, and performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 42 references that are in the map. First posted on arXiv 19 May 2026.

Builds on

21 of 42 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  7. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  8. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  9. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  10. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoningcited by 201
  11. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
  12. 2021Lower Perplexity is Not Always Human-Likecited by 52
9 more
  1. 2024How Does Quantization Affect Multilingual LLMs?cited by 43
  2. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantizationcited by 34
  3. 2024LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkitcited by 33
  4. 2024Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compressioncited by 33
  5. 2024What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbationcited by 23
  6. 2018An Affect-Rich Neural Conversational Model with Biased Attention and Weighted Cross-Entropy Losscited by 21
  7. 2024Quantifying the Capabilities of LLMs across Scale and Precisioncited by 19
  8. 2024Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giantcited by 14
  9. 2024Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolboxcited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Does quantization affect models' performance on long-context tasks?
  2. 2024A Comprehensive Evaluation of Quantization Strategies for Large Language Models
  3. 2024When Quantization Affects Confidence of Large Language Models?
  4. 2025A Comprehensive Evaluation on Quantization Techniques for Large Language Models
  5. 2026Reliability Scaling Laws for Quantized Large Language Models
  6. 2023Understanding the Impact of Post-Training Quantization on Large Language Models
  7. 2025LLM Compression: How Far Can We Go in Balancing Size and Performance?
  8. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms