mlsanityData to 7 Oct 2026

VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts

arXiv 2112.02399 ↗First posted 4 Dec 2021cs.CV cs.CLcited by 42

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Contrastive Language-Image Pre-training (CLIP) has drawn increasing attention recently for its transferable visual representation learning. However, due to the semantic gap within datasets, CLIP's pre-trained image-text alignment becomes sub-optimal on downstream tasks, which severely harms its transferring performance. To better adapt the cross-modality embedding space, we propose to enhance CLIP via Visual-guided Texts, named VT-CLIP. Specifically, we guide textual features of different categories to adaptively explore informative regions on the image and aggregate visual features by attention mechanisms. In this way, the texts become visual-guided, namely, more semantically correlated with downstream images, which greatly benefits the category-wise matching process. In few-shot settings, we evaluate our VT-CLIP on 11 well-known classification datasets to demonstrate its effectiveness.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 44 references that are in the map. First posted on arXiv 4 Dec 2021.

Builds on

33 of 44 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  2. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2017Mask R-CNNcited by 7,259
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
19 more
  1. 2013Describing Textures in the Wildcited by 2,110
  2. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  4. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  5. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  6. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  7. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  8. 2019How Can We Know What Language Models Know?cited by 953
  9. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  10. 2021Domain Generalization: A Surveycited by 760
  11. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  12. 2015Bidirectional LSTM-CRF Models for Sequence Taggingcited by 544
  13. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  14. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  15. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  16. 2019Cross Attention Network for Few-shot Classificationcited by 218
  17. 2021Fast Convergence of DETR with Spatially Modulated Co-Attentioncited by 176
  18. 2018Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answeringcited by 113
  19. 2016Instance-aware Image and Sentence Matching with Selective Multimodal LSTMcited by 43

Built on by

Cited by year

Citing papers per year: 2022: 6; 2023: 14; 2024: 17; 2025: 5.

42 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  3. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  4. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 118
  5. 2022CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attentioncited by 103
  6. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 94
  7. 2023Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinementcited by 84
  8. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  9. 2023ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidancecited by 70
  10. 2022Can Language Understand Depth?cited by 66
  11. 2022PLOT: Prompt Learning with Optimal Transport for Vision-Language Modelscited by 64
  12. 2023Read-only Prompt Optimization for Vision-Language Few-shot Learningcited by 58
30 more
  1. 2023Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-trainingcited by 52
  2. 2022SgVA-CLIP: Semantic-guided Visual Adapting of Vision-Language Models for Few-shot Image Classificationcited by 34
  3. 2024WorDepth: Variational Language Prior for Monocular Depth Estimationcited by 34
  4. 2023Disentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learningcited by 31
  5. 2024Few-shot Adaptation of Multi-modal Foundation Models: A Surveycited by 28
  6. 2024CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Modelscited by 25
  7. 2025Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Modelscited by 23
  8. 2023Tem-adapter: Adapting Image-Text Pretraining for Video Question Answercited by 22
  9. 2024AMU-Tuning: Effective Logit Bias for CLIP-based Few-shot Learningcited by 22
  10. 2024Concept-Guided Prompt Learning for Generalization in Vision-Language Modelscited by 20
  11. 2024RSA: Resolving Scale Ambiguities in Monocular Depth Estimators through Language Descriptionscited by 20
  12. 2023Learning to Adapt CLIP for Few-Shot Monocular Depth Estimationcited by 19
  13. 2023Domain Aligned CLIP for Few-shot Classificationcited by 16
  14. 2024CLIPose: Category-Level Object Pose Estimation with Pre-trained Vision-Language Knowledgecited by 16
  15. 2024Learning to Prompt Segment Anything Modelscited by 13
  16. 2024NeuroBind: Towards Unified Multimodal Representations for Neural Signalscited by 13
  17. 2023Cross-Modal Concept Learning and Inference for Vision-Language Modelscited by 12
  18. 2025Logits DeConfusion with CLIP for Few-Shot Learningcited by 11
  19. 2023Integrating Language-Derived Appearance Elements with Visual Cues in Pedestrian Detectioncited by 10
  20. 2024Conceptual Codebook Learning for Vision-Language Modelscited by 7
  21. 2024Bayesian Exploration of Pre-trained Models for Low-shot Image Classificationcited by 6
  22. 2024Text and Image Are Mutually Beneficial: Enhancing Training-Free Few-Shot Classification with CLIPcited by 6
  23. 2025Training-Free Multi-Style Fusion Through Reference-Based Adaptive Modulationcited by 6
  24. 2024Category-Prompt Refined Feature Learning for Long-Tailed Multi-Label Image Classificationcited by 5
  25. 2023UniBoost: Unsupervised Unimodal Pre-training for Boosting Zero-shot Vision-Language Taskscited by 4
  26. 2023Emotional Theory of Mind: Bridging Fast Visual Processing with Slow Linguistic Reasoningcited by 3
  27. 2024Robust Pedestrian Detection via Constructing Versatile Pedestrian Knowledge Bankcited by 3
  28. 2024NODE-Adapter: Neural Ordinary Differential Equations for Better Vision-Language Reasoningcited by 3
  29. 2025Dual-View Alignment Learning with Hierarchical-Prompt for Class-Imbalance Multi-Label Classificationcited by 2
  30. 2025Dual Prompt Learning for Adapting Vision-Language Models to Downstream Image-Text Retrievalcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models
  2. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
  3. 2024Diffusion Feedback Helps CLIP See Better
  4. 2025Helping CLIP See Both the Forest and the Trees: A Decomposition and Description Approach
  5. 2026Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning
  6. 2025VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captions
  7. 2023Improved baselines for vision-language pre-training
  8. 2023CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancement