# Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models

arXiv: https://arxiv.org/abs/2406.02915 · HTML: https://mlsanity.com/paper/2406.02915/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 42 of its 58 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 30 of 42)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://mlsanity.com/paper/2108.10904/) (2021)
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [Unifying Vision-and-Language Tasks via Text Generation](https://mlsanity.com/paper/2102.02779/) (2021)
- [Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models](https://mlsanity.com/paper/2209.07511/) (2022)
- [Prompt-aligned Gradient for Prompt Tuning](https://mlsanity.com/paper/2205.14865/) (2022)
- [Visual Classification via Description from Large Language Models](https://mlsanity.com/paper/2210.07183/) (2022)
- [Visual-Language Prompt Tuning with Knowledge-guided Context Optimization](https://mlsanity.com/paper/2303.13283/) (2023)
- [Exploring Visual Prompts for Adapting Large-Scale Models](https://mlsanity.com/paper/2203.17274/) (2022)
- [What does a platypus look like? Generating customized prompts for zero-shot image classification](https://mlsanity.com/paper/2209.03320/) (2022)
- [What does CLIP know about a red circle? Visual prompt engineering for VLMs](https://mlsanity.com/paper/2304.06712/) (2023)
- [CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models](https://mlsanity.com/paper/2109.11797/) (2021)
- [Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning](https://mlsanity.com/paper/2308.06038/) (2023)

## Built on by (most cited first, 30 of 30)

- [Enhancing CLIP Robustness via Cross-Modality Alignment](https://mlsanity.com/paper/2510.24038/) (2025)
- [Object Detection with Multimodal Large Vision-Language Models: An In-depth Review](https://mlsanity.com/paper/2508.19294/) (2025)
- [Dynamic Multimodal Prototype Learning in Vision-Language Models](https://mlsanity.com/paper/2507.03657/) (2025)
- [Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editing](https://mlsanity.com/paper/2506.13827/) (2025)
- [Attribute-based Visual Reprogramming for Vision-Language Models](https://mlsanity.com/paper/2501.13982/) (2025)
- [Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrieval](https://mlsanity.com/paper/2512.03276/) (2025)
- [Interpretable Zero-Shot Learning with Locally-Aligned Vision-Language Model](https://mlsanity.com/paper/2506.23822/) (2025)
- [From Local Details to Global Context: Advancing Vision-Language Models with Attention-Based Selection](https://mlsanity.com/paper/2505.13233/) (2025)
- [Vision-Language Models display a strong gender bias](https://mlsanity.com/paper/2508.11262/) (2025)
- [MARE: Multimodal Alignment and Reinforcement for Explainable Deepfake Detection via Vision-Language Models](https://mlsanity.com/paper/2601.20433/) (2026)
- [Interpretable Cross-Domain Few-Shot Learning with Rectified Target-Domain Local Alignment](https://mlsanity.com/paper/2603.17655/) (2026)
- [Generalizable Prompt Learning of CLIP: A Brief Overview](https://mlsanity.com/paper/2503.01263/) (2025)
- [Enhancing Target-unspecific Tasks through a Features Matrix](https://mlsanity.com/paper/2505.03414/) (2025)
- [Image Recognition with Vision and Language Embeddings of VLMs](https://mlsanity.com/paper/2509.09311/) (2025)
- [GeoAlignCLIP: Enhancing Fine-Grained Vision-Language Alignment in Remote Sensing via Multi-Granular Consistency Learning](https://mlsanity.com/paper/2603.09566/) (2026)
- [CARPRT: Class-Aware Zero-Shot Prompt Reweighting for Black-Box Vision-Language Models](https://mlsanity.com/paper/2607.14125/) (2026)
- [Towards Fine-Grained Adaptation of CLIP via a Self-Trained Alignment Score](https://mlsanity.com/paper/2507.09615/) (2025)
- [M3PO: Multimodal-Model-Guided Preference Optimization for Visual Instruction Following](https://mlsanity.com/paper/2508.12458/) (2025)
- [Constrained Prompt Enhancement for Improving Zero-Shot Generalization of Vision-Language Models](https://mlsanity.com/paper/2508.17417/) (2025)
- [A Parameter-Efficient Mixture-of-Experts Framework for Cross-Modal Geo-Localization](https://mlsanity.com/paper/2510.20291/) (2025)
- [Let's Roll a BiFTA: Bi-refinement for Fine-grained Text-visual Alignment in Vision-Language Models](https://mlsanity.com/paper/2601.20419/) (2026)
- [Seeing Through Words: Controlling Visual Retrieval Quality with Language Models](https://mlsanity.com/paper/2602.21175/) (2026)
- [Leveraging Vision-Language Embeddings for Zero-Shot Learning in Histopathology Images](https://mlsanity.com/paper/2503.10731/) (2025)
- [Prototype-Guided Pseudo-Labeling with Neighborhood-Aware Consistency for Unsupervised Adaptation](https://mlsanity.com/paper/2507.22075/) (2025)
- [microCLIP: Unsupervised CLIP Adaptation via Coarse-Fine Token Fusion for Fine-Grained Image Classification](https://mlsanity.com/paper/2510.02270/) (2025)
- [LGCA: Enhancing Semantic Representation via Progressive Expansion](https://mlsanity.com/paper/2511.00419/) (2025)
- [Multi-Granularity Mutual Refinement Network for Zero-Shot Learning](https://mlsanity.com/paper/2511.08163/) (2025)
- [Exploring Weak-to-Strong Generalization for CLIP-based Classification](https://mlsanity.com/paper/2511.18396/) (2025)
- [LAGO: Language-Guided Adaptive Object-Region Focus for Zero-Shot Visual-Text Alignment](https://mlsanity.com/paper/2605.08156/) (2026)
- [CLIP-Guided Label-Free Discriminative Region Scoring for Fine-Grained Classification](https://mlsanity.com/paper/2607.13437/) (2026)

## Abstract (arXiv)

It has recently been discovered that using a pre-trained vision-language model (VLM), e.g., CLIP, to align a whole query image with several finer text descriptions generated by a large language model can significantly enhance zero-shot performance. However, in this paper, we empirically find that the finer descriptions tend to align more effectively with local areas of the query image rather than the whole image, and then we theoretically validate this finding. Thus, we present a method called weighted visual-text cross alignment (WCA). This method begins with a localized visual prompting technique, designed to identify local visual areas within the query image. The local visual areas are then cross-aligned with the finer descriptions by creating a similarity matrix using the pre-trained VLM. To determine how well a query image aligns with each category, we develop a score function based on the weighted similarities in this matrix. Extensive experiments demonstrate that our method significantly improves zero-shot performance across various datasets, achieving results that are even comparable to few-shot learning methods.