mlsanityData to 14 Oct 2026

Perplexity Cost Understates What Activation Quantisation Breaks

arXiv 2609.23125 ↗First posted 19 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Activation quantisation is usually evaluated with an aggregate metric, perplexity, averaged over every token a model predicts. We ask whether that average identifies which computations a quantiser damages. Perplexity turns out to be a reliable aggregate signal: across 12 models from four families and 780 within-model comparisons, the arm perplexity prefers also retains more induction and more retrieval in all but 2.1 and 4.0 percent of cases respectively. But where perplexity has risen by only a factor of 1.2 to 1.5, induction still keeps 0.959 of its intact accuracy while retrieval has already fallen to 0.554, a gap the aggregate number does not surface. This gap has structure, not just size: a matched Gaussian-noise control of the same per-channel magnitude leaves it largely intact, and randomising only the sign of the quantisation error, every magnitude held fixed, is nearly as harmless, so magnitude alone does not explain the damage. Quantising in a rotated basis, which changes coordinate alignment without changing error magnitude, restores induction from 0.001 to 0.980 at three average bits per token in a single-block intervention, though retrieval recovers less completely at the same setting (0.694); end-to-end at four average bits, induction reaches 0.968 and retrieval 0.534. The pattern holds on two further models up to 32B parameters and, in the deployed configurations we tested, under AWQ once activations are pushed to 4 bits. A perplexity target bounds the average cost of a transformation applied to the activation; it does not, by itself, show which computations survived.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 30 references that are in the map. First posted on arXiv 19 Sep 2026.

Builds on

22 of 30 references.

References · 22

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  6. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  7. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  9. 2022Toy Models of Superpositioncited by 971
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  11. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  12. 2024Scaling and evaluating sparse autoencoderscited by 553
10 more
  1. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  2. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  3. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  4. 2024Massive Activations in Large Language Modelscited by 294
  5. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 116
  6. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 62
  7. 2021Grounding Representation Similarity with Statistical Testingcited by 29
  8. 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Modelscited by 21
  9. 2026Perplexity Can Miss SAE Feature Damage Under Quantizationcited by 5
  10. 2026Decomposing MXFP4 quantization error for LLM reinforcement learning: reducible bias, recoverable deadzone, and an irreducible floorcited by 1

Built on by

Not cited yet.