Cosine Similarity Is Not Evidence: Measuring the Noise Floor of Interpretability Transfer Under Quantization

A statistic reported without the quantity needed to interpret it is not evidence. We develop that thesis for a concrete practice in AI safety. Interpretability artifacts are calibrated on full-precision weights, deployed on quantized ones, and certified as surviving the change by scale-invariant statistics (cosine similarity, correlation, AUROC) that are reported without their noise floor. For the difference-in-means direction estimator, the split-half floor is governed by one dimensionless number, kappa = nρ2/d. The closed form mathbbE[cos] ≈ (1+4/kappa)-1 is classical; the missing input is the class separation ρ, which we measure on real activations; no compression-transfer study we know of reports it. On Qwen2.5-1.5B-Instruct, ρ = 33--61 across depth, so two independent runs of the estimator agree to 0.978--0.994 by sampling alone. A published cosine of 0.996 between full-precision and quantized refusal directions therefore cannot be read as preservation without the n it was computed at, which is not reported. Where n is known, we judge each low-bit cosine against the split-half null measured within that quantized model, because a full-precision null assumes the low-bit estimator has the same variance. That assumption is exactly what a null exists to test. The result is plain: at INT4 the direction rotated, and the deficit exceeds the estimator's own noise. At INT8 we detect no movement, which is not an equivalence claim. We also show that a scale-invariant statistic cannot distinguish translation from attenuation of a transferred decision variable, although the two call for opposite remedies. We close with reporting recommendations that cost one forward pass. Code, data, and a one-cell reproduction are released at https://github.com/pvarshh/quantinterp