# Bridge the Modality and Capability Gaps in Vision-Language Model Selection

arXiv: https://arxiv.org/abs/2403.13797 · HTML: https://mlsanity.com/paper/2403.13797/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 1 of its 64 references that are in the map.
- First posted on arXiv 20 Mar 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 1 of 1)

- [Revisiting Meta-Learning as Supervised Learning](https://mlsanity.com/paper/2002.00573/) (2020)

## Built on by (most cited first, 20 of 20)

- [Number it: Temporal Grounding Videos like Flipping Manga](https://mlsanity.com/paper/2411.10332/) (2024)
- [Parrot: Multilingual Visual Instruction Tuning](https://mlsanity.com/paper/2406.02539/) (2024)
- [Wings: Learning Multimodal LLMs without Text-only Forgetting](https://mlsanity.com/paper/2406.03496/) (2024)
- [Mimic In-Context Learning for Multimodal Tasks](https://mlsanity.com/paper/2504.08851/) (2025)
- [LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Model](https://mlsanity.com/paper/2604.02097/) (2026)
- [Image-to-Text Logic Jailbreak: Your Imagination can Help You Do Anything](https://mlsanity.com/paper/2407.02534/) (2024)
- [Data Selection for Fine-tuning Vision Language Models via Cross Modal Alignment Trajectories](https://mlsanity.com/paper/2510.01454/) (2025)
- [Learning Modal-Mixed Chain-of-Thought Reasoning with Latent Embeddings](https://mlsanity.com/paper/2602.00574/) (2026)
- [Concept-as-Tree: A Controllable Synthetic Data Framework Makes Stronger Personalized VLMs](https://mlsanity.com/paper/2503.12999/) (2025)
- [Bridging the Gap in Vision Language Models in Identifying Unsafe Concepts Across Modalities](https://mlsanity.com/paper/2507.11155/) (2025)
- [Representations of Text and Images Align From Layer One](https://mlsanity.com/paper/2601.08017/) (2026)
- [VisTIRA: Closing the Image-Text Modality Gap in Visual Math Reasoning via Structured Tool Integration](https://mlsanity.com/paper/2601.14440/) (2026)
- [Learning to Rank Pre-trained Vision-Language Models for Downstream Tasks](https://mlsanity.com/paper/2412.20682/) (2024)
- [Mordal: Automated Pretrained Model Selection for Vision Language Models](https://mlsanity.com/paper/2502.00241/) (2025)
- [Model Specific Task Similarity for Vision Language Model Selection via Layer Conductance](https://mlsanity.com/paper/2602.01346/) (2026)
- [XPERT: Expert Knowledge Transfer for Effective Training of Language Models](https://mlsanity.com/paper/2605.08842/) (2026)
- [Plug-and-Adapt: Multimodal Coreference Resolution at First Sight with a Pretrained Alignment Model](https://mlsanity.com/paper/2606.17950/) (2026)
- [Visually Grounded Self-Reflection for Vision-Language Models via Reinforcement Learning](https://mlsanity.com/paper/2607.02490/) (2026)
- [From Inference to Adaptation: A Unified Optimal Transport View of Vision Language Model](https://mlsanity.com/paper/2608.18339/) (2026)
- [To Trust or Not to Trust: Retrieval-Augmented Fact Checking in Speech](https://mlsanity.com/paper/2609.30227/) (2026)

## Abstract (arXiv)

Vision Language Models (VLMs) excel in zero-shot image classification by pairing images with textual category names. The expanding variety of Pre-Trained VLMs enhances the likelihood of identifying a suitable VLM for specific tasks. To better reuse the VLM resource and fully leverage its potential on different zero-shot image classification tasks, a promising strategy is selecting appropriate Pre-Trained VLMs from the VLM Zoo, relying solely on the text data of the target dataset without access to the dataset's images. In this paper, we analyze two inherent challenges in assessing the ability of a VLM in this Language-Only VLM selection: the "Modality Gap" - the disparity in VLM's embeddings across two different modalities, making text a less reliable substitute for images; and the "Capability Gap" - the discrepancy between the VLM's overall ranking and its ranking for target dataset, hindering direct prediction of a model's dataset-specific performance from its general performance. We propose VLM Selection With gAp Bridging (SWAB) to mitigate the negative impact of two gaps. SWAB first adopts optimal transport to capture the relevance between open-source and target datasets with a transportation matrix. It then uses this matrix to transfer useful statistics of VLMs from open-source datasets to the target dataset for bridging two gaps. By bridging two gaps to obtain better substitutes for test images, SWAB can accurately predict the performance ranking of different VLMs on the target task without the need for the dataset's images. Experiments across various VLMs and image classification datasets validate SWAB's effectiveness.