mlsanityData to 14 Oct 2026

Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models

arXiv 2406.02915 ↗First posted 5 Jun 2024cs.CV cs.LGcited by 30

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

It has recently been discovered that using a pre-trained vision-language model (VLM), e.g., CLIP, to align a whole query image with several finer text descriptions generated by a large language model can significantly enhance zero-shot performance. However, in this paper, we empirically find that the finer descriptions tend to align more effectively with local areas of the query image rather than the whole image, and then we theoretically validate this finding. Thus, we present a method called weighted visual-text cross alignment (WCA). This method begins with a localized visual prompting technique, designed to identify local visual areas within the query image. The local visual areas are then cross-aligned with the finer descriptions by creating a similarity matrix using the pre-trained VLM. To determine how well a query image aligns with each category, we develop a score function based on the weighted similarities in this matrix. Extensive experiments demonstrate that our method significantly improves zero-shot performance across various datasets, achieving results that are even comparable to few-shot learning methods.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 58 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

42 of 58 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning11 Aug
  2. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Concepts12 Jun
  3. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMs13 Apr
  4. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimization23 Mar
  5. 2022Unified Vision and Language Prompt Learning13 Oct
  6. 2022Visual Classification via Description from Large Language Models13 Oct
  7. 2022Learning to Decompose Visual Features with Latent Textual Prompts9 Oct
  8. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  9. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  10. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  11. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models24 Sep
  12. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Segment Anythingcited by 8,234
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2013Describing Textures in the Wildcited by 2,125
  7. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  8. 2022Visual Prompt Tuningcited by 1,681
  9. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  10. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  11. 2019Natural Adversarial Examplescited by 1,408
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
18 more
  1. 2023PaLM 2 Technical Reportcited by 1,162
  2. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  3. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  4. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  5. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  6. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  7. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  8. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 274
  9. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 267
  10. 2022AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilitiescited by 85
  11. 2023Fine-Grained Visual Promptingcited by 82
  12. 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-trainingcited by 67
  13. 2022Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learningcited by 57
  14. 2022OrdinalCLIP: Learning Rank Prompts for Language-Guided Ordinal Regressioncited by 44
  15. 2023π-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolationcited by 40
  16. 2022Visual Commonsense in Pretrained Unimodal and Multimodal Modelscited by 39
  17. 2022Progressive Video Summarization via Multimodal Self-supervised Learningcited by 21
  18. 2023On the Difference of BERT-style and CLIP-style Text Encoderscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 22; 2026: 8.

30 papers.

Sort:

By area:

  1. 2025Enhancing CLIP Robustness via Cross-Modality Alignmentcited by 15
  2. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  3. 2025Dynamic Multimodal Prototype Learning in Vision-Language Modelscited by 13
  4. 2025Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editingcited by 10
  5. 2025Attribute-based Visual Reprogramming for Vision-Language Modelscited by 8
  6. 2025Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrievalcited by 7
  7. 2025Interpretable Zero-Shot Learning with Locally-Aligned Vision-Language Modelcited by 6
  8. 2025From Local Details to Global Context: Advancing Vision-Language Models with Attention-Based Selectioncited by 5
  9. 2025Vision-Language Models display a strong gender biascited by 5
  10. 2026MARE: Multimodal Alignment and Reinforcement for Explainable Deepfake Detection via Vision-Language Modelscited by 4
  11. 2026Interpretable Cross-Domain Few-Shot Learning with Rectified Target-Domain Local Alignmentcited by 4
  12. 2025Generalizable Prompt Learning of CLIP: A Brief Overviewcited by 3
18 more
  1. 2025Enhancing Target-unspecific Tasks through a Features Matrixcited by 2
  2. 2025Image Recognition with Vision and Language Embeddings of VLMscited by 2
  3. 2026GeoAlignCLIP: Enhancing Fine-Grained Vision-Language Alignment in Remote Sensing via Multi-Granular Consistency Learningcited by 2
  4. 2026CARPRT: Class-Aware Zero-Shot Prompt Reweighting for Black-Box Vision-Language Modelscited by 2
  5. 2025Towards Fine-Grained Adaptation of CLIP via a Self-Trained Alignment Scorecited by 1
  6. 2025M3PO: Multimodal-Model-Guided Preference Optimization for Visual Instruction Followingcited by 1
  7. 2025Constrained Prompt Enhancement for Improving Zero-Shot Generalization of Vision-Language Modelscited by 1
  8. 2025A Parameter-Efficient Mixture-of-Experts Framework for Cross-Modal Geo-Localizationcited by 1
  9. 2026Let's Roll a BiFTA: Bi-refinement for Fine-grained Text-visual Alignment in Vision-Language Modelscited by 1
  10. 2026Seeing Through Words: Controlling Visual Retrieval Quality with Language Modelscited by 1
  11. 2025Leveraging Vision-Language Embeddings for Zero-Shot Learning in Histopathology Imagescited by 0
  12. 2025Prototype-Guided Pseudo-Labeling with Neighborhood-Aware Consistency for Unsupervised Adaptationcited by 0
  13. 2025microCLIP: Unsupervised CLIP Adaptation via Coarse-Fine Token Fusion for Fine-Grained Image Classificationcited by 0
  14. 2025LGCA: Enhancing Semantic Representation via Progressive Expansioncited by 0
  15. 2025Multi-Granularity Mutual Refinement Network for Zero-Shot Learningcited by 0
  16. 2025Exploring Weak-to-Strong Generalization for CLIP-based Classificationcited by 0
  17. 2026LAGO: Language-Guided Adaptive Object-Region Focus for Zero-Shot Visual-Text Alignmentcited by 0
  18. 2026CLIP-Guided Label-Free Discriminative Region Scoring for Fine-Grained Classificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment
  2. 2024ComAlign: Compositional Alignment in Vision-Language Models
  3. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts
  4. 2025Language-Image Alignment with Fixed Text Encoders
  5. 2025AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding
  6. 2024Better Language Models Exhibit Higher Visual Alignment
  7. 2022X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks
  8. 2024Learning to Rank Pre-trained Vision-Language Models for Downstream Tasks