# Whitening Inverts the Hierarchy: What the Norm of a Whitened Embedding Measures

arXiv: https://arxiv.org/abs/2609.23117 · HTML: https://mlsanity.com/paper/2609.23117/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 19 Sep 2026.

## Builds on (most cited first, 14 of 14)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Out-of-Distribution Detection with Deep Nearest Neighbors](https://mlsanity.com/paper/2204.06507/) (2022)
- [Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss](https://mlsanity.com/paper/2106.04156/) (2021)
- [Whitening Sentence Representations for Better Semantics and Faster Retrieval](https://mlsanity.com/paper/2103.15316/) (2021)
- [Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods](https://mlsanity.com/paper/2205.11508/) (2022)
- [The Double-Ellipsoid Geometry of CLIP](https://mlsanity.com/paper/2411.14517/) (2024)
- [Whitened CLIP as a Likelihood Surrogate of Images and Captions](https://mlsanity.com/paper/2505.06934/) (2025)
- [InfoNCE Induces Gaussian Distribution](https://mlsanity.com/paper/2602.24012/) (2026)
- [Training-free Detection of Generated Videos via Spatial-Temporal Likelihoods](https://mlsanity.com/paper/2603.15026/) (2026)
- [General and Domain-Specific Zero-shot Detection of Generated Images via Conditional Likelihood](https://mlsanity.com/paper/2512.05590/) (2025)
- [The Universal Normal Embedding](https://mlsanity.com/paper/2603.21786/) (2026)

## Abstract (arXiv)

Whitening a foundation-model embedding and using its squared norm as a training-free likelihood surrogate is motivated by the observation that whitened coordinates often appear approximately standard normal. We show that this observation follows from the projection central limit theorem and therefore does not imply a Gaussian joint distribution. Across multiple encoders and three training objectives, we find systematic over-dispersion of the whitened radius relative to the Gaussian reference, including against distributional clones with identical mean and covariance. We further show that the commonly reported agreement between empirical and theoretical norm statistics is an algebraic consequence of in-sample whitening and does not constitute evidence for Gaussianity. We identify the mechanism behind this behavior: whitening reverses the encoder's spectral hierarchy, shifting the contribution to the squared norm toward near-degenerate directions that encode predominantly noise. In these directions, the dominant variability is governed by a single input-dependent scale. We estimate this scale from two moments and use it to predict, without additional free parameters, the cross-dependence between disjoint spectral halves. These results indicate that the squared whitened norm is better interpreted as a Mahalanobis measure of semantic atypicality than as a log-likelihood. This interpretation explains both its practical effectiveness and its calibration failures: the statistic can rank and detect atypical samples consistently with nonparametric density estimates and across encoders trained with different objectives, while Gaussian tail thresholds can be inaccurate by orders of magnitude. etc.