mlsanityData to 7 Oct 2026

CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models

arXiv 2109.11797 ↗First posted 24 Sep 2021cs.CV cs.CLcited by 182

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Pre-Trained Vision-Language Models (VL-PTMs) have shown promising capabilities in grounding natural language in image data, facilitating a broad variety of cross-modal tasks. However, we note that there exists a significant gap between the objective forms of model pre-training and fine-tuning, resulting in a need for large amounts of labeled data to stimulate the visual grounding capability of VL-PTMs for downstream tasks. To address the challenge, we present Cross-modal Prompt Tuning (CPT, alternatively, Colorful Prompt Tuning), a novel paradigm for tuning VL-PTMs, which reformulates visual grounding into a fill-in-the-blank problem with color-based co-referential markers in image and text, maximally mitigating the gap. In this way, CPT enables strong few-shot and even zero-shot visual grounding capabilities of VL-PTMs. Comprehensive experimental results show that the prompt-tuned VL-PTMs outperform their fine-tuned counterparts by a large margin (e.g., 17.3% absolute accuracy improvement, and 73.8% relative standard deviation reduction on average with one shot in RefCOCO evaluation). We make the data and code for this paper publicly available at https://github.com/thunlp/CPT.

Key facts. Cited by 182 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 56 of its 68 references that are in the map. First posted on arXiv 24 Sep 2021.

Builds on

56 of 68 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 55

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2017Mask R-CNNcited by 7,259
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2015VQA: Visual Question Answeringcited by 2,489
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
38 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  2. 2019Language Models as Knowledge Bases?cited by 1,922
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  4. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  6. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  7. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  8. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  9. 2016Modeling Context in Referring Expressionscited by 1,054
  10. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  11. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  12. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  13. 2019How Can We Know What Language Models Know?cited by 953
  14. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  15. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  16. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  17. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  18. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  19. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  20. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  21. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  22. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  23. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 413
  24. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  25. 2016Visual Dialogcited by 377
  26. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  27. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  28. 2018Learning to Compose Dynamic Tree Structures for Visual Contextscited by 258
  29. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 246
  30. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  31. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  32. 2019Knowledge-Embedded Routing Network for Scene Graph Generationcited by 163
  33. 2021Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Generationcited by 137
  34. 2019Learning by Abstraction: The Neural State Machinecited by 113
  35. 2015Grounding of Textual Phrases in Images by Reconstructioncited by 112
  36. 2018Explainable and Explicit Visual Reasoning over Scene Graphscited by 103
  37. 2017Grounding Referring Expressions in Images by Variational Contextcited by 82
  38. 2019Zero-Shot Grounding of Objects from Natural Language Queriescited by 79

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 49; 2023: 60; 2024: 43; 2025: 24; 2026: 4.

182 papers.

Sort:

By area:

  1. 2022Visual Prompt Tuningcited by 1,674
  2. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  3. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  4. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  5. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  6. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  7. 2022Multimodal Learning with Transformers: A Surveycited by 399
  8. 2023Segment Anything in High Qualitycited by 373
  9. 2022Prompt-aligned Gradient for Prompt Tuningcited by 339
  10. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 307
  11. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  12. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
38 more
  1. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  2. 2022A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunitiescited by 162
  3. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  4. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 150
  5. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  6. 2022Unified Vision and Language Prompt Learningcited by 145
  7. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  8. 2022Backbone is All Your Need: A Simplified Architecture for Visual Object Trackingcited by 119
  9. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 119
  10. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  11. 2023In-Context Learning Unlocked for Diffusion Modelscited by 103
  12. 2022DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotationscited by 102
  13. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 98
  14. 2023Fine-Grained Visual Promptingcited by 81
  15. 2022Visual Prompt Tuning for Generative Transfer Learningcited by 75
  16. 2023Prompting for Multimodal Hateful Meme Classificationcited by 68
  17. 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65
  18. 2022Contrastive Adapters for Foundation Model Group Robustnesscited by 62
  19. 2024Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Trainingcited by 62
  20. 2022Prompting for Multi-Modal Trackingcited by 58
  21. 2022Fine-Grained Scene Graph Generation with Data Transfercited by 57
  22. 2023Diversity-Aware Meta Visual Promptingcited by 57
  23. 2024Toward Visual Grounding: A Surveycited by 52
  24. 2022Dual Modality Prompt Tuning for Vision-Language Pre-Trained Modelcited by 51
  25. 2022Texts as Images in Prompt Tuning for Multi-Label Image Recognitioncited by 51
  26. 2022ADAPT: Vision-Language Navigation with Modality-Aligned Action Promptscited by 45
  27. 2024DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLMcited by 45
  28. 2022Towards Visual-Prompt Temporal Answering Grounding in Medical Instructional Videocited by 43
  29. 2023Panoptic Scene Graph Generation with Semantics-Prototype Learningcited by 42
  30. 2022Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Modelscited by 40
  31. 2022SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samplescited by 39
  32. 2022Visual Commonsense in Pretrained Unimodal and Multimodal Modelscited by 39
  33. 2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
  34. 2022The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoningcited by 38
  35. 2024SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusioncited by 36
  36. 2022Modular and Parameter-Efficient Multimodal Fusion with Promptingcited by 35
  37. 2023CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Groundingcited by 35
  38. 2022Pseudo-Q: Generating Pseudo Language Queries for Visual Groundingcited by 33

50 most cited of 182 All 182 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023LaViP:Language-Grounded Visual Prompts
  2. 2024Can Better Text Semantics in Prompt Tuning Improve VLM Generalization?
  3. 2023Distribution-Aware Prompt Tuning for Vision-Language Models
  4. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs
  5. 2025Towards Understanding Visual Grounding in Visual Language Models
  6. 2023Language Models as Black-Box Optimizers for Vision-Language Models
  7. 2024Concept-Guided Prompt Learning for Generalization in Vision-Language Models
  8. 2023Towards Efficient Vision-Language Tuning: More Information Density, More Generalizability