# Cosine Similarity Is Not Evidence: Measuring the Noise Floor of Interpretability Transfer Under Quantization

arXiv: https://arxiv.org/abs/2609.30275 · HTML: https://mlsanity.com/paper/2609.30275/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 17 references that are in the map.
- First posted on arXiv 12 Aug 2026.

## Builds on (most cited first, 11 of 11)

- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [Massive Activations in Large Language Models](https://mlsanity.com/paper/2402.17762/) (2024)
- [Understanding (Un)Reliability of Steering Vectors in Language Models](https://mlsanity.com/paper/2505.22637/) (2025)
- [Semantics at an Angle: When Cosine Similarity Works Until It Doesn't](https://mlsanity.com/paper/2504.16318/) (2025)
- [Why Do Some Inputs Break Low-Bit LLM Quantization?](https://mlsanity.com/paper/2506.12044/) (2025)
- [Towards Understanding and Improving Refusal in Compressed Models via Mechanistic Interpretability](https://mlsanity.com/paper/2504.04215/) (2025)
- [Perplexity Can Miss SAE Feature Damage Under Quantization](https://mlsanity.com/paper/2606.03002/) (2026)
- [Do Activation Monitors Survive Model Updates? Benchmarking, Predicting, and Repairing Activation-Monitor Staleness](https://mlsanity.com/paper/2606.15980/) (2026)
- [What Cosine Similarity of Label Representations Can and Cannot Tell us](https://mlsanity.com/paper/2603.29488/) (2026)
- [Quality Is Not a Safety Proxy Under Quantization](https://mlsanity.com/paper/2606.10154/) (2026)

## Abstract (arXiv)

A statistic reported without the quantity needed to interpret it is not evidence. We develop that thesis for a concrete practice in AI safety. Interpretability artifacts are calibrated on full-precision weights, deployed on quantized ones, and certified as surviving the change by scale-invariant statistics (cosine similarity, correlation, AUROC) that are reported without their noise floor. For the difference-in-means direction estimator, the split-half floor is governed by one dimensionless number, kappa = nρ2/d. The closed form mathbbE[cos] ≈ (1+4/kappa)-1 is classical; the missing input is the class separation ρ, which we measure on real activations; no compression-transfer study we know of reports it. On Qwen2.5-1.5B-Instruct, ρ = 33--61 across depth, so two independent runs of the estimator agree to 0.978--0.994 by sampling alone. A published cosine of 0.996 between full-precision and quantized refusal directions therefore cannot be read as preservation without the n it was computed at, which is not reported. Where n is known, we judge each low-bit cosine against the split-half null measured within that quantized model, because a full-precision null assumes the low-bit estimator has the same variance. That assumption is exactly what a null exists to test. The result is plain: at INT4 the direction rotated, and the deficit exceeds the estimator's own noise. At INT8 we detect no movement, which is not an equivalence claim. We also show that a scale-invariant statistic cannot distinguish translation from attenuation of a transferred decision variable, although the two call for opposite remedies. We close with reporting recommendations that cost one forward pass. Code, data, and a one-cell reproduction are released at https://github.com/pvarshh/quantinterp