mlsanityData to 7 Oct 2026

Your CLIP has 164 dimensions of noise: Exploring the embeddings covariance eigenspectrum of contrastively pretrained vision-language transformers

arXiv 2605.14893 ↗First posted 14 May 2026cs.CV cs.AI cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Contrastively pre-trained Vision-Language Models (VLMs) serve as powerful feature extractors. Yet, their shared latent spaces are prone to structural anomalies and act as repositories for non-semantic, multi-modal noise. To address this phenomenon, we employ spectral decomposition of covariance matrices to decompose the VLM latent space into a multi-modal semantic signal component and a shared noise subspace. We observe that this noise geometry exhibits strong subgroup invariance across distinct data subsets. Crucially, pruning these shared noise dimensions is mainly harmless, preserving or actively improving downstream task performance. By isolating true semantic signals from artifactual noise, this work provides new mechanistic insights into the representational structure of modern VLMs, suggesting that a substantial fraction of their latent geometry is governed by shared, architecture-level noise rather than task-relevant semantics alone.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 33 references that are in the map. First posted on arXiv 14 May 2026.

Builds on

29 of 33 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  10. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  11. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  12. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
17 more
  1. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  2. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  3. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  4. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
  5. 2025Open Problems in Mechanistic Interpretabilitycited by 173
  6. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 151
  7. 2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
  8. 2025Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversioncited by 42
  9. 2025Shared Global and Local Geometry of Language Model Embeddingscited by 34
  10. 2025ILIAS: Instance-Level Image retrieval At Scalecited by 29
  11. 2024It's Not a Modality Gap: Characterizing and Addressing the Contrastive Gapcited by 26
  12. 2025Whitened CLIP as a Likelihood Surrogate of Images and Captionscited by 16
  13. 2024Scendi Score: Prompt-Aware Diversity Evaluation via Schur Complement of CLIP Embeddingscited by 15
  14. 2025Enhancing CLIP Robustness via Cross-Modality Alignmentcited by 14
  15. 2026Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Modelscited by 10
  16. 2026Is the Modality Gap a Bug or a Feature? A Robustness Perspectivecited by 4
  17. 2026IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignmentcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Through the LENS: Local Geometric Decomposition of Vision-Language Model Representations
  2. 2026Cross-Modal Redundancy and the Geometry of Vision-Language Embeddings
  3. 2026The Universal Normal Embedding
  4. 2026Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance
  5. 2025Unlearning the Noisy Correspondence Makes CLIP More Robust
  6. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models
  7. 2026One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models
  8. 2025Evaluating Robustness of Vision-Language Models Under Noisy Conditions