Reevaluating the Intra-Modal Misalignment Hypothesis in CLIP
Not on the map. Why?
Abstract (arXiv)
Recent research suggested that the embeddings produced by CLIP-like contrastive language-image training are suboptimal for image-only tasks. The main theory is that the inter-modal (language-image) alignment loss ignores intra-modal (image-image) alignment, leading to poorly calibrated distances between images. In this study, we question this intra-modal misalignment hypothesis. We reexamine its foundational theoretical argument, the indicators used to support it, and the performance metrics affected. For the theoretical argument, we demonstrate that there are no such supposed degrees of freedom for image embedding distances. For the empirical measures, our findings reveal they yield similar results for language-image trained models (CLIP, SigLIP) and image-image trained models (DINO, SigLIP2). This indicates the observed phenomena do not stem from a misalignment specific to the former. Experiments on the commonly studied intra-modal tasks retrieval and few-shot classification confirm that addressing task ambiguity, not supposed misalignment, is key for best results.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 42 of its 52 references that are in the map. First posted on arXiv 17 Mar 2026.
Builds on
42 of 52 references.
References · 42
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
- 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
- 2021Learning to Prompt for Vision-Language Modelscited by 2,269
- 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
- 2013Describing Textures in the Wildcited by 2,110
- 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
- 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
- 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
- 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
30 more
- 2019A Closer Look at Few-shot Classificationcited by 679
- 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
- 2021Extract Free Dense Labels from CLIPcited by 580
- 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
- 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
- 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
- 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
- 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
- 2023Scaling Open-Vocabulary Object Detectioncited by 335
- 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
- 2018Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarkingcited by 189
- 2023Zero-Shot Composed Image Retrieval with Textual Inversioncited by 171
- 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 118
- 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
- 2023Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinementcited by 84
- 2023AVID: Any-Length Video Inpainting with Diffusion Modelcited by 81
- 2023Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learningcited by 53
- 2024A Hard-to-Beat Baseline for Training-free CLIP-based Adaptationcited by 51
- 2025Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversioncited by 42
- 2023Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIPcited by 27
- 2024Mitigate the Gap: Investigating Approaches for Improving Cross-Modal Alignment in CLIPcited by 23
- 2024Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classificationcited by 18
- 2025Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stagescited by 17
- 2025ProKeR: A Kernel Perspective on Few-Shot Adaptation of Large Vision-Language Modelscited by 15
- 2025Enhancing CLIP Robustness via Cross-Modality Alignmentcited by 14
- 2025Logits DeConfusion with CLIP for Few-Shot Learningcited by 11
- 2024Understanding Retrieval-Augmented Task Adaptation for Vision-Language Modelscited by 10
- 2025Project-Probe-Aggregate: Efficient Fine-Tuning for Group Robustnesscited by 9
- 2024Towards flexible perception with visual memorycited by 8
- 2025SeMoBridge: Semantic Modality Bridge for Efficient Few-Shot Adaptation of CLIPcited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.