mlsanityData to 7 Oct 2026

How Divergence Becomes Decision Flips in Compressed Language Models

arXiv 2610.00694 ↗First posted 30 Sep 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Compression reports summarize how far a compressed language model moved from the dense one, usually by a KL divergence; a deployment that relies on the dense model's outputs needs to know how many of its decisions changed. We show that total variation, not KL, answers this directly. Across 802 compressed and perturbed copies of 19 open models on five corpora and nine mechanically unrelated perturbation families, the rate at which the arg-max token changes (the flip rate) tracks total variation at a ratio with median 1.05, with no fitted constant. KL converts into flips only through its square root and a factor that varies fourfold across models and corpora, because KL averages over tokens before the root is taken; first-order statistics averaged per token, such as Hellinger distance, avoid this, but reports rarely give them. As a result, of two compressors reported on different models and corpora whose flip rates differ by at least 10%, KL assigns the smaller divergence to the one that changes more decisions in 11% of cases, total variation in 1%. Two pre-registered tests mark the limits: on a held-out code corpus the ratio held for all eight models while three predictions about KL each failed for half of them or more, and on three new models with real kernels it stayed in its band for 37 of 38 checkpoints but fell below one on code for two models. In vLLM speculative decoding, total variation measured under teacher forcing predicts greedy draft acceptance with a mean relative error of 1.1--2.4%, without the task-specific calibration that KL needs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 50 of its 62 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

50 of 62 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  11. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  12. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
38 more
  1. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  2. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  3. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  4. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  5. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  6. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  7. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  8. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  9. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  10. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  11. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  12. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  13. 2014New insights and perspectives on the natural gradient methodcited by 435
  14. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  15. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  16. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  17. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  18. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  19. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  20. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  21. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  22. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  23. 2019What Do Compressed Deep Neural Networks Forget?cited by 164
  24. 2024An empirical study of LLaMA3 quantization: from LLMs to MLLMscited by 63
  25. 2020Positive-Congruent Training: Towards Regression-Free Model Updatescited by 55
  26. 2021Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compressioncited by 38
  27. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantizationcited by 34
  28. 2021On the Reproducibility of Neural Network Predictionscited by 28
  29. 2023Tailoring Language Generation Models under Total Variation Distancecited by 27
  30. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 26
  31. 2024Direct Alignment of Draft Model for Speculative Decoding with Chat-Fine-Tuned LLMscited by 21
  32. 2024QSpec: Speculative Decoding with Complementary Quantization Schemescited by 18
  33. 2025AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoderscited by 9
  34. 2025Establishing a Scale for Kullback-Leibler Divergence in Language Models Across Various Settingscited by 3
  35. 2026The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMscited by 3
  36. 2026Displacement Is Not Direction: Evaluating Fidelity Metrics for Quantized LLM Deploymentcited by 2
  37. 2026Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Modelscited by 1
  38. 2026Accuracy is Not Enough: A Divergence-Based Approach to Evaluate Fidelity Loss in Quantized LLMscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression
  2. 2024Language Models Resist Alignment: Evidence From Data Compression
  3. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models
  4. 2026Every Time I Hire a Linguist, Inference Costs Go Down: On Linguistic Rules as Effective Prompt Compressors
  5. 2026Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels
  6. 2025A Probabilistic Perspective on Model Collapse
  7. 2026When Summaries Distort Decisions: Information Fidelity in LLM-Compressed Financial Analysis
  8. 2026A Theoretical Analysis of Why Masked Diffusion Models Mitigate the Reversal Curse