mlsanityData to 14 Oct 2026

Closing the Modality Gap for Mixed Modality Search

arXiv 2507.19054 ↗First posted 25 Jul 2025cs.CV cs.AI cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Mixed modality search -- retrieving information across a heterogeneous corpus composed of images, texts, and multimodal documents -- is an important yet underexplored real-world application. In this work, we investigate how contrastive vision-language models, such as CLIP, perform on the mixed modality search task. Our analysis reveals a critical limitation: these models exhibit a pronounced modality gap in the embedding space, where image and text embeddings form distinct clusters, leading to intra-modal ranking bias and inter-modal fusion failure. To address this issue, we propose GR-CLIP, a lightweight post-hoc calibration method that removes the modality gap in CLIP's embedding space. Evaluated on MixBench -- the first benchmark specifically designed for mixed modality search -- GR-CLIP improves NDCG@10 by up to 26 percentage points over CLIP, surpasses recent vision-language generative embedding models by 4 percentage points, while using 75x less compute.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 28 of its 38 references that are in the map. First posted on arXiv 25 Jul 2025.

Builds on

28 of 38 references.

References · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  9. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  10. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  11. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  12. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
16 more
  1. 2020Fact or Fiction: Verifying Scientific Claimscited by 554
  2. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  3. 2018Stacked Cross Attention for Image-Text Matchingcited by 416
  4. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  5. 2019Clotho: An Audio Captioning Datasetcited by 358
  6. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 337
  7. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
  8. 2023Demystifying CLIP Datacited by 270
  9. 2023Multimodal Deep Learningcited by 258
  10. 2024Generative Representational Instruction Tuningcited by 233
  11. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 228
  12. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 195
  13. 2020Visual News: Benchmark and Challenges in News Image Captioningcited by 121
  14. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 113
  15. 2023Diagnosing and Rectifying Vision Models using Languagecited by 58
  16. 2024Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Datacited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 6.

12 papers.

  1. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 18
  2. 2025Guided Query Refinement: Multimodal Hybrid Retrieval with Test-Time Optimizationcited by 12
  3. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 3
  4. 2025When Embedding Models Meet: Procrustes Bounds and Applicationscited by 2
  5. 2026TextME: Bridging Unseen Modalities Through Text Descriptionscited by 2
  6. 2025Text-Only Training for Image Captioning with Retrieval Augmentation and Modality Gap Correctioncited by 1
  7. 2026Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledgecited by 1
  8. 2025Adapting MLLMs for Nuanced Video Retrievalcited by 0
  9. 2026The Geometry of Compromise: Unlocking Generative Capabilities via Controllable Modality Alignmentcited by 0
  10. 2026COMET: Concept Space Dissection of the Modality Gap in Audio-Text Multimodal Contrastive Embeddingscited by 0
  11. 2026NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inferencecited by 0
  12. 2026One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion
  2. 2025Post-pre-training for Modality Alignment in Vision-Language Foundation Models
  3. 2025Fill the Gap: Quantifying and Reducing the Modality Gap in Image-Text Representation Learning
  4. 2025UniMoCo: Unified Modality Completion for Robust Multi-Modal Embeddings
  5. 2022A Comprehensive Empirical Study of Vision-Language Pre-trained Model for Supervised Cross-Modal Retrieval
  6. 2026Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning
  7. 2025Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptions
  8. 2024ModalChorus: Visual Probing and Alignment of Multi-modal Embeddings via Modal Fusion Map