# The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs

arXiv: https://arxiv.org/abs/2607.08734 · HTML: https://mlsanity.com/paper/2607.08734/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 29 references that are in the map.
- First posted on arXiv 9 Jul 2026.

## Builds on (most cited first, 23 of 23)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression](https://mlsanity.com/paper/2306.03078/) (2023)
- [SqueezeLLM: Dense-and-Sparse Quantization](https://mlsanity.com/paper/2306.07629/) (2023)
- [Do Large Language Models Know What They Don't Know?](https://mlsanity.com/paper/2305.18153/) (2023)
- [A Comprehensive Evaluation of Quantization Strategies for Large Language Models](https://mlsanity.com/paper/2402.16775/) (2024)
- [Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning](https://mlsanity.com/paper/2501.03035/) (2025)
- ["Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization](https://mlsanity.com/paper/2411.02355/) (2024)
- [Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency](https://mlsanity.com/paper/2504.03360/) (2025)
- [LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit](https://mlsanity.com/paper/2405.06001/) (2024)
- [Accuracy is Not All You Need](https://mlsanity.com/paper/2407.09141/) (2024)
- [Through the Lens of Core Competency: Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2308.07902/) (2023)
- [Benchmarking Post-Training Quantization of Large Language Models under Microscaling Floating Point Formats](https://mlsanity.com/paper/2601.09555/) (2026)
- [CORECODE: A Common Sense Annotated Dialogue Dataset with Benchmark Tasks for Chinese Large Language Models](https://mlsanity.com/paper/2312.12853/) (2023)
- [Quantized Large Language Models in Biomedical Natural Language Processing: Evaluation and Recommendation](https://mlsanity.com/paper/2509.04534/) (2025)
- [AQUA-LLM: Evaluating Accuracy, Quantization, and Adversarial Robustness Trade-offs in LLMs for Cybersecurity Question Answering](https://mlsanity.com/paper/2509.13514/) (2025)

## Built on by (most cited first, 4 of 4)

- [Certifying Compressed Language Models: An Audit and a Statistical Toolkit](https://mlsanity.com/paper/2608.15046/) (2026)
- [Which Decisions Low-Bit Quantization Breaks, and How to Predict Them](https://mlsanity.com/paper/2608.06564/) (2026)
- [Accuracy is Not Enough: A Divergence-Based Approach to Evaluate Fidelity Loss in Quantized LLMs](https://mlsanity.com/paper/2609.07664/) (2026)
- [How Divergence Becomes Decision Flips in Compressed Language Models](https://mlsanity.com/paper/2610.00694/) (2026)

## Abstract (arXiv)

Post-Training Quantization has become widely used to compress large language models to make them deployable on resource-constrained devices. However, the evaluation of quantization methods mainly uses accuracy and perplexity, which cannot capture the behavioral changes in the quantized variants. In this work, we propose Correctness Agreement, a decision-level metric that can measure the intersection of correct predictions between the base model and its quantized variant. We use this metric across multiple models and quantization bit levels (8-bit to 2-bit), and we find that the base and quantized variants usually have a shift in behavior even when accuracy and perplexity are preserved. In order to explain this effect, we study the effect of quantization on the structure of the attention weights using statistical and distributional measures. The results reveal a breakpoint at low bit widths and show that query and key projections are more sensitive to quantization than the value and output projections. These results prove the illusion of equivalency between the base and quantized models and inspire behavioral evaluation beyond perplexity and accuracy for quantization methods.