# Displacement Is Not Direction: Evaluating Fidelity Metrics for Quantized LLM Deployment

arXiv: https://arxiv.org/abs/2606.19558 · HTML: https://mlsanity.com/paper/2606.19558/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 17 Jun 2026.

## Builds on (most cited first, 14 of 14)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Compressing LLMs: The Truth is Rarely Pure and Never Simple](https://mlsanity.com/paper/2310.01382/) (2023)
- ["Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization](https://mlsanity.com/paper/2411.02355/) (2024)
- [MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Design](https://mlsanity.com/paper/2412.14590/) (2024)
- [BitPruning: Learning Bitlengths for Aggressive and Accurate Quantization](https://mlsanity.com/paper/2002.03090/) (2020)
- [Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giant](https://mlsanity.com/paper/2409.11055/) (2024)
- [GSM8K-V: Can Vision Language Models Solve Grade School Math Word Problems in Visual Contexts](https://mlsanity.com/paper/2509.25160/) (2025)
- [Divergent Token Metrics: Measuring degradation to prune away LLM components -- and optimize quantization](https://mlsanity.com/paper/2311.01544/) (2023)

## Built on by (most cited first, 3 of 3)

- [Certifying Compressed Language Models: An Audit and a Statistical Toolkit](https://mlsanity.com/paper/2608.15046/) (2026)
- [Tied Trit-Planes: Constraining PTQTP to a Uniform Nine-Level Quantizer, with a Persistent Folded Format for Disk-Streamed Mixture-of-Experts Serving](https://mlsanity.com/paper/2608.08910/) (2026)
- [How Divergence Becomes Decision Flips in Compressed Language Models](https://mlsanity.com/paper/2610.00694/) (2026)

## Abstract (arXiv)

Fidelity metrics, such as per-token KL divergence (KLD) against a high-precision reference, are often used in practice as low-cost proxies for benchmark quality. We test this practice on a 28-quant cohort of Qwen3.6-35B-A3B and a 41-quant cohort of Devstral-Small-2-24B, evaluated across a suite of downstream benchmarks. We find that KLD is strongly correlated with benchmark score over the full cohort (ρ=-0.72 on Qwen and ρ=-0.86 on Devstral, both with p<0.001). However, this relationship collapses to non-significance in the near-baseline silent zone (ρ=+0.00 on Qwen and ρ=-0.24, p=0.36, on Devstral). This collapse persists across 14 measurement variants, including different KLD aggregations, perplexity formulations, top-1 agreement, calibration corpora, and context lengths. At the per-prompt level, KLD has only weak failure-prediction power on code, with failed-vs-passed geometric-mean ratios in [1.08,1.22] across five models on LiveCodeBench, and fails as a cross-model router, achieving only 42.3%-49.4% accuracy on disagreement prompts. We trace the collapse to a structural decomposition: KLD primarily measures the volume of disagreement with the reference, with silent-zone composite ρ=+0.94 (p<0.001) on Qwen and +0.55 (p=0.03) on Devstral, while its relationship to the direction of those disagreements is weak and task-conditional.