mlsanityData to 14 Oct 2026

Whitened CLIP as a Likelihood Surrogate of Images and Captions

arXiv 2505.06934 ↗First posted 11 May 2025eess.IV cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

Likelihood approximations for images are not trivial to compute and can be useful in many applications. We examine the use of Contrastive Language-Image Pre-training (CLIP) to assess the likelihood of images and captions. We introduce textitWhitened CLIP, a novel transformation of the CLIP latent space via an invertible linear operation. This transformation ensures that each feature in the embedding space has zero mean, unit standard deviation, and no correlation with all other features, resulting in an identity covariance matrix. We show that the whitened embeddings statistics can be well approximated as a standard normal distribution, thus, the log-likelihood is estimated simply by the square Euclidean norm in the whitened embedding space. The whitening procedure is completely training-free and performed using a pre-computed whitening matrix, hence, is very fast. We present several preliminary experiments demonstrating the properties and applicability of these likelihood scores to images and captions.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 28 of its 54 references that are in the map. First posted on arXiv 11 May 2025.

Builds on

28 of 54 references.

References · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2022Generative Adversarial Networkscited by 5,627
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  11. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  12. 2019Natural Adversarial Examplescited by 1,408
16 more
  1. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  2. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  3. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  4. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  5. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  6. 2021SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Modelscited by 415
  7. 2023Implicit Diffusion Models for Continuous Super-Resolutioncited by 218
  8. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 152
  9. 2023Hyperbolic Image-Text Representationscited by 136
  10. 2019Deep Learning on Image Denoising: An overviewcited by 128
  11. 2019Image inpainting: A reviewcited by 89
  12. 2023ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmarkcited by 77
  13. 2024The Double-Ellipsoid Geometry of CLIPcited by 42
  14. 2024SynArtifact: Classifying and Alleviating Artifacts in Synthetic Images via Vision-Language Modelcited by 15
  15. 2024Embedding Geometries of Contrastive Language-Image Pre-Trainingcited by 6
  16. 2024Energy-Based Models with Applications to Speech and Language Processingcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 13.

16 papers.

Sort:

By area:

  1. 2026InfoNCE Induces Gaussian Distributioncited by 16
  2. 2026Training-free Detection of Generated Videos via Spatial-Temporal Likelihoodscited by 8
  3. 2025Training-Free Policy Violation Detection via Activation-Space Whitening in LLMscited by 6
  4. 2026The Universal Normal Embeddingcited by 3
  5. 2026Epistemic Uncertainty Quantification for Pre-trained VLMs via Riemannian Flow Matchingcited by 2
  6. 2026SCoCCA: Multi-modal Sparse Concept Decomposition via Canonical Correlation Analysiscited by 1
  7. 2025CLIP-like Model as a Foundational Density Ratio Estimatorcited by 0
  8. 2025Diffusion Models for Low-Light Image Enhancement: A Multi-Perspective Taxonomy and Performance Analysiscited by 0
  9. 2026Make it SING: Analyzing Semantic Invariants in Classifierscited by 0
  10. 2026GeoFlowVLM: Geometry-Aware Joint Uncertainty for Frozen Vision-Language Embeddingcited by 0
  11. 2026Your CLIP has 164 dimensions of noise: Exploring the embeddings covariance eigenspectrum of contrastively pretrained vision-language transformerscited by 0
  12. 2026Beyond Templates: Revisiting Zero-Shot Remote Sensing through Meta-Promptingcited by 0
4 more
  1. 2026A Classifier-Agnostic Zero-Shot Adversarial Attack Detection via CLIPcited by 0
  2. 2026The Hyperspherical Geometry of CLIP Latent Space: A Semantic Mixture Modelcited by 0
  3. 2026DEFUSE: Generalizable Backdoor Defense for Self-Supervised Encoders with Generative Priorscited by 0
  4. 2026Whitening Inverts the Hierarchy: What the Norm of a Whitened Embedding Measurescited by 0