mlsanityData to 7 Oct 2026

Learning to Prompt for Vision-Language Models

arXiv 2109.01134 ↗First posted 2 Sep 2021cs.CV cs.AI cs.LGcited by 2,269

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Large pre-trained vision-language models like CLIP have shown great potential in learning representations that are transferable across a wide range of downstream tasks. Different from the traditional representation learning that is based mostly on discretized labels, vision-language pre-training aligns images and texts in a common feature space, which allows zero-shot transfer to a downstream task via prompting, i.e., classification weights are synthesized from natural language describing classes of interest. In this work, we show that a major challenge for deploying such models in practice is prompt engineering, which requires domain expertise and is extremely time-consuming -- one needs to spend a significant amount of time on words tuning since a slight change in wording could have a huge impact on performance. Inspired by recent advances in prompt learning research in natural language processing (NLP), we propose Context Optimization (CoOp), a simple approach specifically for adapting CLIP-like vision-language models for downstream image recognition. Concretely, CoOp models a prompt's context words with learnable vectors while the entire pre-trained parameters are kept fixed. To handle different image recognition tasks, we provide two implementations of CoOp: unified context and class-specific context. Through extensive experiments on 11 datasets, we demonstrate that CoOp requires as few as one or two shots to beat hand-crafted prompts with a decent margin and is able to gain significant improvements over prompt engineering with more shots, e.g., with 16 shots the average gain is around 15% (with the highest reaching over 45%). Despite being a learning-based approach, CoOp achieves superb domain generalization performance compared with the zero-shot model using hand-crafted prompts.

Key facts. Cited by 2,269 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 61 references that are in the map. First posted on arXiv 2 Sep 2021.

Builds on

37 of 61 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
25 more
  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2013Describing Textures in the Wildcited by 2,110
  3. 2019Language Models as Knowledge Bases?cited by 1,922
  4. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  5. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  7. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  8. 2019Natural Adversarial Examplescited by 1,401
  9. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  10. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  11. 2019How Can We Know What Language Models Know?cited by 953
  12. 2021Domain Generalization: A Surveycited by 760
  13. 2021GPT Understands, Toocited by 715
  14. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  15. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  16. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  17. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 435
  18. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  19. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  20. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 246
  21. 2013Zero-Shot Learning Through Cross-Modal Transfercited by 237
  22. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  23. 2016Learning Visual N-Grams from Web Datacited by 95
  24. 2015Predicting Deep Zero-Shot Convolutional Neural Networks using Textual Descriptionscited by 64
  25. 2017Self-supervised learning of visual features through embedding images into text topic spacescited by 43

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 173; 2023: 482; 2024: 627; 2025: 628; 2026: 339.

2,269 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  3. 2022Visual Prompt Tuningcited by 1,674
  4. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  5. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  6. 2021Grounded Language-Image Pre-trainingcited by 1,147
  7. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  8. 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 974
  9. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  10. 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
  11. 2021Robust fine-tuning of zero-shot modelscited by 785
  12. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
38 more
  1. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  2. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  3. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  4. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  5. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  6. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  7. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  8. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  9. 2021Image Segmentation Using Text and Image Promptscited by 509
  10. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
  11. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  12. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  13. 2023LangSplat: 3D Language Gaussian Splattingcited by 410
  14. 2022Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)cited by 396
  15. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  16. 2023Segment Anything in High Qualitycited by 373
  17. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
  18. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 355
  19. 2022Prompt-aligned Gradient for Prompt Tuningcited by 339
  20. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 327
  21. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  22. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  23. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  24. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  25. 2023WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentationcited by 311
  26. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 310
  27. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  28. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 307
  29. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 296
  30. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
  31. 2023Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspectivecited by 276
  32. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  33. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  34. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  35. 2023AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detectioncited by 270
  36. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 266
  37. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  38. 2023Generating Images with Multimodal Language Modelscited by 259

50 most cited of 2,269 All 2,269 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models
  2. 2024PLPP: Prompt Learning with Perplexity Is Self-Distillation for Vision-Language Models
  3. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications
  4. 2024Concept-Guided Prompt Learning for Generalization in Vision-Language Models
  5. 2024Make Prompts Adaptable: Bayesian Modeling for Vision-Language Prompt Learning with Data-Dependent Prior
  6. 2026Joint Semantic Token Selection and Prompt Optimization for Interpretable Prompt Learning
  7. 2024Generalizable Prompt Tuning for Vision-Language Models
  8. 2024SEP: Self-Enhanced Prompt Tuning for Visual-Language Model