mlsanityData to 7 Oct 2026

Visual-Language Prompt Tuning with Knowledge-guided Context Optimization

arXiv 2303.13283 ↗First posted 23 Mar 2023cs.CV cs.CLcited by 307

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Prompt tuning is an effective way to adapt the pre-trained visual-language model (VLM) to the downstream task using task-related textual tokens. Representative CoOp-based work combines the learnable textual tokens with the class tokens to obtain specific textual knowledge. However, the specific textual knowledge is the worse generalization to the unseen classes because it forgets the essential general textual knowledge having a strong generalization ability. To tackle this issue, we introduce a novel Knowledge-guided Context Optimization (KgCoOp) to enhance the generalization ability of the learnable prompt for unseen classes. The key insight of KgCoOp is that forgetting about essential knowledge can be alleviated by reducing the discrepancy between the learnable prompt and the hand-crafted prompt. Especially, KgCoOp minimizes the discrepancy between the textual embeddings generated by learned prompts and the hand-crafted prompts. Finally, adding the KgCoOp upon the contrastive loss can make a discriminative prompt for both seen and unseen tasks. Extensive evaluation of several benchmarks demonstrates that the proposed Knowledge-guided Context Optimization is an efficient method for prompt tuning, i.e., achieves better performance with less training time.

Key facts. Cited by 307 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 46 references that are in the map. First posted on arXiv 23 Mar 2023.

Builds on

32 of 46 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022Unified Vision and Language Prompt Learning13 Oct
  2. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  3. 2022Prompt Distribution Learning6 May
  4. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  5. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  6. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models24 Sep
  7. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2013Describing Textures in the Wildcited by 2,110
13 more
  1. 2019Language Models as Knowledge Bases?cited by 1,922
  2. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  3. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  4. 2022Visual Prompt Tuningcited by 1,674
  5. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  6. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  8. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  9. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  10. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  11. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  12. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  13. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 26; 2024: 86; 2025: 127; 2026: 68.

307 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2023Repurposing Diffusion-Based Image Generators for Monocular Depth Estimationcited by 323
  3. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 103
  4. 2024C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersioncited by 82
  5. 2025Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysiscited by 78
  6. 2023Consistency-guided Prompt Learning for Vision-Language Modelscited by 73
  7. 2023A Closer Look at the Few-Shot Adaptation of Large Vision-Language Modelscited by 67
  8. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?cited by 65
  9. 2024LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIPcited by 60
  10. 2023DePT: Decoupled Prompt Tuningcited by 58
  11. 2024A Hard-to-Beat Baseline for Training-free CLIP-based Adaptationcited by 51
  12. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
38 more
  1. 2023EventCLIP: Adapting CLIP for Event-based Object Recognitioncited by 36
  2. 2024Boosting Vision-Language Models with Transductioncited by 36
  3. 2023EventBind: Learning a Unified Representation to Bind Them All for Event-based Open-world Understandingcited by 33
  4. 2024DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learningcited by 32
  5. 2025Bayesian Prompt Flow Learning for Zero-Shot Anomaly Detectioncited by 31
  6. 2024Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Modelscited by 30
  7. 2024Visual Fourier Prompt Tuningcited by 30
  8. 2024Few-shot Adaptation of Multi-modal Foundation Models: A Surveycited by 28
  9. 2024Robust Calibration of Large Vision-Language Adapterscited by 28
  10. 2024BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Modelscited by 28
  11. 2024AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportationcited by 27
  12. 2025Localized Concept Erasure for Text-to-Image Diffusion Models Using Training-Free Gated Low-Rank Adaptationcited by 27
  13. 2024Any-Shift Prompting for Generalization over Distributionscited by 24
  14. 2024Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Modelscited by 23
  15. 2025Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Modelscited by 23
  16. 2024Few-shot Adaptation of Medical Vision-Language Modelscited by 22
  17. 2023HGCLIP: Exploring Vision-Language Models with Graph Representations for Hierarchical Understandingcited by 21
  18. 2024Image-Text Co-Decomposition for Text-Supervised Semantic Segmentationcited by 21
  19. 2024Historical Test-time Prompt Tuning for Vision Foundation Modelscited by 21
  20. 2024Concept-Guided Prompt Learning for Generalization in Vision-Language Modelscited by 20
  21. 2024XCoOp: Explainable Prompt Learning for Computer-Aided Diagnosis via Concept-guided Context Optimizationcited by 20
  22. 2024Advancing Textual Prompt Learning with Anchored Attributescited by 20
  23. 2023APoLLo: Unified Adapter and Prompt Learning for Vision Language Modelscited by 19
  24. 2025A Survey on Efficient Vision-Language Modelscited by 19
  25. 2023Learning Domain-Aware Detection Head with Prompt Tuningcited by 18
  26. 2024Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillationcited by 18
  27. 2024Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalizationcited by 17
  28. 2024CLIP with Generative Latent Replay: a Strong Baseline for Incremental Learningcited by 17
  29. 2024Deep Correlated Prompting for Visual Recognition with Missing Modalitiescited by 17
  30. 2025DPC: Dual-Prompt Collaboration for Tuning Vision-Language Modelscited by 17
  31. 2023Text-driven Prompt Generation for Vision-Language Models in Federated Learningcited by 16
  32. 2024Visual-Augmented Dynamic Semantic Prototype for Generative Zero-Shot Learningcited by 16
  33. 2024M2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learningcited by 16
  34. 2025PromptGuard: Soft Prompt-Guided Unsafe Content Moderation for Text-to-Image Modelscited by 16
  35. 2025Reliable Few-shot Learning under Dual Noisescited by 16
  36. 2025Concept Pinpoint Eraser for Text-to-image Diffusion Models via Residual Attention Gatecited by 16
  37. 2024Open-Vocabulary Calibration for Fine-tuned CLIPcited by 15
  38. 2024Few-Shot Recognition via Stage-Wise Retrieval-Augmented Finetuningcited by 15

50 most cited of 307 All 307 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024SEP: Self-Enhanced Prompt Tuning for Visual-Language Model
  2. 2023Knowledge-Aware Prompt Tuning for Generalizable Vision-Language Models
  3. 2023TCP:Textual-based Class-aware Prompt tuning for Visual-Language Model
  4. 2022Multitask Vision-Language Prompt Tuning
  5. 2024Fully Fine-tuned CLIP Models are Efficient Few-Shot Learners
  6. 2025GA2-CLIP: Generic Attribute Anchor for Efficient Prompt Tuningin Video-Language Models
  7. 2023Distribution-Aware Prompt Tuning for Vision-Language Models
  8. 2023Context-Aware Prompt Tuning for Vision-Language Model with Dual-Alignment