# Whitened CLIP as a Likelihood Surrogate of Images and Captions

arXiv: https://arxiv.org/abs/2505.06934 · HTML: https://mlsanity.com/paper/2505.06934/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 54 references that are in the map.
- First posted on arXiv 11 May 2025.

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models](https://mlsanity.com/paper/2104.14951/) (2021)
- [Implicit Diffusion Models for Continuous Super-Resolution](https://mlsanity.com/paper/2303.16491/) (2023)
- [Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models](https://mlsanity.com/paper/2402.14848/) (2024)
- [Hyperbolic Image-Text Representations](https://mlsanity.com/paper/2304.09172/) (2023)
- [Deep Learning on Image Denoising: An overview](https://mlsanity.com/paper/1912.13171/) (2019)
- [Image inpainting: A review](https://mlsanity.com/paper/1909.06399/) (2019)
- [ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark](https://mlsanity.com/paper/2303.13648/) (2023)
- [The Double-Ellipsoid Geometry of CLIP](https://mlsanity.com/paper/2411.14517/) (2024)
- [SynArtifact: Classifying and Alleviating Artifacts in Synthetic Images via Vision-Language Model](https://mlsanity.com/paper/2402.18068/) (2024)
- [Embedding Geometries of Contrastive Language-Image Pre-Training](https://mlsanity.com/paper/2409.13079/) (2024)
- [Energy-Based Models with Applications to Speech and Language Processing](https://mlsanity.com/paper/2403.10961/) (2024)

## Built on by (most cited first, 16 of 16)

- [InfoNCE Induces Gaussian Distribution](https://mlsanity.com/paper/2602.24012/) (2026)
- [Training-free Detection of Generated Videos via Spatial-Temporal Likelihoods](https://mlsanity.com/paper/2603.15026/) (2026)
- [Training-Free Policy Violation Detection via Activation-Space Whitening in LLMs](https://mlsanity.com/paper/2512.03994/) (2025)
- [The Universal Normal Embedding](https://mlsanity.com/paper/2603.21786/) (2026)
- [Epistemic Uncertainty Quantification for Pre-trained VLMs via Riemannian Flow Matching](https://mlsanity.com/paper/2601.21662/) (2026)
- [SCoCCA: Multi-modal Sparse Concept Decomposition via Canonical Correlation Analysis](https://mlsanity.com/paper/2603.13884/) (2026)
- [CLIP-like Model as a Foundational Density Ratio Estimator](https://mlsanity.com/paper/2506.22881/) (2025)
- [Diffusion Models for Low-Light Image Enhancement: A Multi-Perspective Taxonomy and Performance Analysis](https://mlsanity.com/paper/2510.05976/) (2025)
- [Make it SING: Analyzing Semantic Invariants in Classifiers](https://mlsanity.com/paper/2603.14610/) (2026)
- [GeoFlowVLM: Geometry-Aware Joint Uncertainty for Frozen Vision-Language Embedding](https://mlsanity.com/paper/2605.13352/) (2026)
- [Your CLIP has 164 dimensions of noise: Exploring the embeddings covariance eigenspectrum of contrastively pretrained vision-language transformers](https://mlsanity.com/paper/2605.14893/) (2026)
- [Beyond Templates: Revisiting Zero-Shot Remote Sensing through Meta-Prompting](https://mlsanity.com/paper/2606.20702/) (2026)
- [A Classifier-Agnostic Zero-Shot Adversarial Attack Detection via CLIP](https://mlsanity.com/paper/2606.30342/) (2026)
- [The Hyperspherical Geometry of CLIP Latent Space: A Semantic Mixture Model](https://mlsanity.com/paper/2607.13660/) (2026)
- [DEFUSE: Generalizable Backdoor Defense for Self-Supervised Encoders with Generative Priors](https://mlsanity.com/paper/2608.25851/) (2026)
- [Whitening Inverts the Hierarchy: What the Norm of a Whitened Embedding Measures](https://mlsanity.com/paper/2609.23117/) (2026)

## Abstract (arXiv)

Likelihood approximations for images are not trivial to compute and can be useful in many applications. We examine the use of Contrastive Language-Image Pre-training (CLIP) to assess the likelihood of images and captions. We introduce textitWhitened CLIP, a novel transformation of the CLIP latent space via an invertible linear operation. This transformation ensures that each feature in the embedding space has zero mean, unit standard deviation, and no correlation with all other features, resulting in an identity covariance matrix. We show that the whitened embeddings statistics can be well approximated as a standard normal distribution, thus, the log-likelihood is estimated simply by the square Euclidean norm in the whitened embedding space. The whitening procedure is completely training-free and performed using a pre-computed whitening matrix, hence, is very fast. We present several preliminary experiments demonstrating the properties and applicability of these likelihood scores to images and captions.