mlsanityData to 7 Oct 2026

Conditional Prompt Learning for Vision-Language Models

arXiv 2203.05557 ↗First posted 10 Mar 2022cs.CV cs.AI cs.CLcited by 1,427

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

With the rise of powerful pre-trained vision-language models like CLIP, it becomes essential to investigate ways to adapt these models to downstream datasets. A recently proposed method named Context Optimization (CoOp) introduces the concept of prompt learning -- a recent trend in NLP -- to the vision domain for adapting pre-trained vision-language models. Specifically, CoOp turns context words in a prompt into a set of learnable vectors and, with only a few labeled images for learning, can achieve huge improvements over intensively-tuned manual prompts. In our study we identify a critical problem of CoOp: the learned context is not generalizable to wider unseen classes within the same dataset, suggesting that CoOp overfits base classes observed during training. To address the problem, we propose Conditional Context Optimization (CoCoOp), which extends CoOp by further learning a lightweight neural network to generate for each image an input-conditional token (vector). Compared to CoOp's static prompts, our dynamic prompts adapt to each instance and are thus less sensitive to class shift. Extensive experiments show that CoCoOp generalizes much better than CoOp to unseen classes, even showing promising transferability beyond a single dataset; and yields stronger domain generalization performance as well. Code is available at https://github.com/KaiyangZhou/CoOp.

Key facts. Cited by 1,427 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 67 references that are in the map. First posted on arXiv 10 Mar 2022.

Builds on

48 of 67 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  2. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models24 Sep
  3. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
33 more
  1. 2013Describing Textures in the Wildcited by 2,110
  2. 2019Language Models as Knowledge Bases?cited by 1,922
  3. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  4. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  5. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  6. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  7. 2019Natural Adversarial Examplescited by 1,401
  8. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  9. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  10. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  11. 2019How Can We Know What Language Models Know?cited by 953
  12. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  13. 2021Robust fine-tuning of zero-shot modelscited by 785
  14. 2021Domain Generalization: A Surveycited by 760
  15. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  16. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  17. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  18. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  19. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  20. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  21. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  22. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  23. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  24. 2017Zero-Shot Learning -- The Good, the Bad and the Uglycited by 254
  25. 2013Zero-Shot Learning Through Cross-Modal Transfercited by 237
  26. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  27. 2018Zero-shot Recognition via Semantic Embeddings and Knowledge Graphscited by 155
  28. 2016An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wildcited by 146
  29. 2016Learning Visual N-Grams from Web Datacited by 95
  30. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 88
  31. 2015Predicting Deep Zero-Shot Convolutional Neural Networks using Textual Descriptionscited by 64
  32. 2017Self-supervised learning of visual features through embedding images into text topic spacescited by 43
  33. 2022Exploring Hierarchical Graph Representation for Large-Scale Zero-Shot Image Classificationcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 85; 2023: 331; 2024: 410; 2025: 388; 2026: 213.

1,427 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  3. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  4. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  5. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  6. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  7. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
  8. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  9. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  10. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  11. 2022Prompt-aligned Gradient for Prompt Tuningcited by 339
  12. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 327
38 more
  1. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  2. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 310
  3. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 307
  4. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  5. 2023AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detectioncited by 270
  6. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  7. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  8. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  9. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  10. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  11. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  12. 2023Teaching CLIP to Count to Tencited by 194
  13. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192
  14. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 189
  15. 2022Patching open-vocabulary models by interpolating weightscited by 187
  16. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 186
  17. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 183
  18. 2023Forgery-aware Adaptive Transformer for Generalizable Synthetic Image Detectioncited by 180
  19. 2022Fine-tuned CLIP Models are Efficient Video Learnerscited by 178
  20. 2022CLIP-ReID: Exploiting Vision-Language Model for Image Re-Identification without Concrete Text Labelscited by 177
  21. 2023Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrievalcited by 155
  22. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  23. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  24. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  25. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 150
  26. 2023Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuningcited by 150
  27. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 149
  28. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  29. 2022Unified Vision and Language Prompt Learningcited by 145
  30. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  31. 2024Efficient Test-Time Adaptation of Vision-Language Modelscited by 138
  32. 2022Neural Prompt Searchcited by 133
  33. 2023CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say Nocited by 130
  34. 2024AdaCLIP: Adapting CLIP with Hybrid Learnable Prompts for Zero-Shot Anomaly Detectioncited by 126
  35. 2022Unsupervised Prompt Learning for Vision-Language Modelscited by 125
  36. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  37. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Modelscited by 124
  38. 2022Task Residual for Tuning Vision-Language Modelscited by 122

50 most cited of 1,427 All 1,427 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Concept-Guided Prompt Learning for Generalization in Vision-Language Models
  2. 2025Accelerating Conditional Prompt Learning via Masked Image Modeling for Vision-Language Models
  3. 2024PLPP: Prompt Learning with Perplexity Is Self-Distillation for Vision-Language Models
  4. 2025Cluster-Aware Prompt Ensemble Learning for Few-Shot Vision-Language Model Adaptation
  5. 2024Make Prompts Adaptable: Bayesian Modeling for Vision-Language Prompt Learning with Data-Dependent Prior
  6. 2024Towards Compatible Fine-tuning for Vision-Language Model Updates
  7. 2026Criterion-Conditional In-Context Learning: Evaluating Criterion-Shift Adaptation in Vision-Language Models
  8. 2026Joint Semantic Token Selection and Prompt Optimization for Interpretable Prompt Learning