mlsanityData to 14 Oct 2026

The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs

arXiv 2607.08734 ↗First posted 9 Jul 2026cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Post-Training Quantization has become widely used to compress large language models to make them deployable on resource-constrained devices. However, the evaluation of quantization methods mainly uses accuracy and perplexity, which cannot capture the behavioral changes in the quantized variants. In this work, we propose Correctness Agreement, a decision-level metric that can measure the intersection of correct predictions between the base model and its quantized variant. We use this metric across multiple models and quantization bit levels (8-bit to 2-bit), and we find that the base and quantized variants usually have a shift in behavior even when accuracy and perplexity are preserved. In order to explain this effect, we study the effect of quantization on the structure of the attention weights using statistical and distributional measures. The results reveal a breakpoint at low bit widths and show that query and key projections are more sensitive to quantization than the value and output projections. These results prove the illusion of equivalency between the base and quantized models and inspire behavioral evaluation beyond perplexity and accuracy for quantization methods.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 23 of its 29 references that are in the map. First posted on arXiv 9 Jul 2026.

Builds on

23 of 29 references.

References · 23

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  9. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  10. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  11. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  12. 2023Do Large Language Models Know What They Don't Know?cited by 243
11 more
  1. 2024A Comprehensive Evaluation of Quantization Strategies for Large Language Modelscited by 48
  2. 2025Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoningcited by 37
  3. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantizationcited by 34
  4. 2025Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latencycited by 34
  5. 2024LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkitcited by 33
  6. 2024Accuracy is Not All You Needcited by 31
  7. 2023Through the Lens of Core Competency: Survey on Evaluation of Large Language Modelscited by 12
  8. 2026Benchmarking Post-Training Quantization of Large Language Models under Microscaling Floating Point Formatscited by 10
  9. 2023CORECODE: A Common Sense Annotated Dialogue Dataset with Benchmark Tasks for Chinese Large Language Modelscited by 3
  10. 2025Quantized Large Language Models in Biomedical Natural Language Processing: Evaluation and Recommendationcited by 3
  11. 2025AQUA-LLM: Evaluating Accuracy, Quantization, and Adversarial Robustness Trade-offs in LLMs for Cybersecurity Question Answeringcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Certifying Compressed Language Models: An Audit and a Statistical Toolkitcited by 2
  2. 2026Which Decisions Low-Bit Quantization Breaks, and How to Predict Themcited by 1
  3. 2026Accuracy is Not Enough: A Divergence-Based Approach to Evaluate Fidelity Loss in Quantized LLMscited by 1
  4. 2026How Divergence Becomes Decision Flips in Compressed Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels
  2. 2024What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbation
  3. 2026Quantization Degradation in Large Language Models: A Signal-Noise Perspective
  4. 2026The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally
  5. 2025Does quantization affect models' performance on long-context tasks?
  6. 2025Quantized but Deceptive? A Multi-Dimensional Truthfulness Evaluation of Quantized LLMs
  7. 2024When Quantization Affects Confidence of Large Language Models?
  8. 2024Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giant