mlsanityData to 14 Oct 2026

What does CLIP know about a red circle? Visual prompt engineering for VLMs

arXiv 2304.06712 ↗First posted 13 Apr 2023cs.CVcited by 218

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Large-scale Vision-Language Models, such as CLIP, learn powerful image-text representations that have found numerous applications, from zero-shot classification to text-to-image generation. Despite that, their capabilities for solving novel discriminative tasks via prompting fall behind those of large language models, such as GPT-3. Here we explore the idea of visual prompt engineering for solving computer vision tasks beyond classification by editing in image space instead of text. In particular, we discover an emergent ability of CLIP, where, by simply drawing a red circle around an object, we can direct the model's attention to that region, while also maintaining global information. We show the power of this simple approach by achieving state-of-the-art in zero-shot referring expressions comprehension and strong performance in keypoint localization tasks. Finally, we draw attention to some potential ethical concerns of large language-vision models.

Key facts. Cited by 218 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 58 of its 70 references that are in the map. First posted on arXiv 13 Apr 2023.

Builds on

58 of 70 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Texts as Images in Prompt Tuning for Multi-Label Image Recognition23 Nov
  2. 2022Multitask Vision-Language Prompt Tuning21 Nov
  3. 2022Unified Vision and Language Prompt Learning13 Oct
  4. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  5. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models24 Sep
  6. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 52

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2022Visual Prompt Tuningcited by 1,681
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  11. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  12. 2016Modeling Context in Referring Expressionscited by 1,063
38 more
  1. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  2. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  3. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  4. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  5. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  6. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  7. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  8. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  9. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  10. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  11. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  12. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  13. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 310
  14. 2022Visual Prompting via Image Inpaintingcited by 250
  15. 2021Fake It Till You Make It: Face analysis in the wild using synthetic data alonecited by 200
  16. 2022Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localizationcited by 150
  17. 2023Debiasing Vision-Language Models via Biased Promptscited by 149
  18. 2019SPair-71k: A Large-scale Benchmark for Semantic Correspondencecited by 146
  19. 2021Referring Transformer: A One-step Approach to Multi-task Visual Groundingcited by 143
  20. 2018Learning to Assemble Neural Module Tree Networks for Visual Groundingcited by 140
  21. 2021Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implicationscited by 128
  22. 2015Natural Language Object Retrievalcited by 124
  23. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 120
  24. 2022TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cutcited by 113
  25. 2019Dynamic Graph Attention for Referring Expression Comprehensioncited by 111
  26. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  27. 2022A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learningcited by 104
  28. 2018Neighbourhood Watch: Referring Expression Comprehension via Language-guided Graph Attention Networkscited by 102
  29. 2021Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Searchcited by 90
  30. 2022PromptCAL: Contrastive Affinity Learning via Auxiliary Prompts for Generalized Novel Category Discoverycited by 87
  31. 2022Disentangling visual and written concepts in CLIPcited by 58
  32. 2022Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learningcited by 57
  33. 2017Comprehension-guided referring expressionscited by 52
  34. 2019Learning Cross-modal Context Graph for Visual Groundingcited by 49
  35. 2019Referring Expression Object Segmentation with Caption-Aware Consistencycited by 46
  36. 2022ProposalCLIP: Unsupervised Open-Category Object Proposal Generation via Exploiting CLIP Cuescited by 42
  37. 2022Pseudo-Q: Generating Pseudo Language Queries for Visual Groundingcited by 33
  38. 2021Discriminative Triad Matching and Reconstruction for Weakly Referring Expression Groundingcited by 21

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 22; 2024: 70; 2025: 79; 2026: 47.

218 papers.

Sort:

By area:

  1. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  2. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  3. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  4. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  5. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 268
  6. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 190
  7. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 166
  8. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 161
  9. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 151
  10. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 147
  11. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  12. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
38 more
  1. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  2. 2023In-Context Learning Unlocked for Diffusion Modelscited by 104
  3. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 101
  4. 2023Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videoscited by 86
  5. 2023Fine-Grained Visual Promptingcited by 82
  6. 2024pix2gestalt: Amodal Segmentation by Synthesizing Wholescited by 70
  7. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavorcited by 69
  8. 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65
  9. 2024PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Modelscited by 62
  10. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 61
  11. 2023Segment Any Anomaly without Training via Hybrid Prompt Regularizationcited by 60
  12. 2024Multi-Object Hallucination in Vision-Language Modelscited by 60
  13. 2023ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulationcited by 59
  14. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Modelscited by 57
  15. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  16. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 54
  17. 2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 54
  18. 2023VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolutioncited by 52
  19. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policycited by 52
  20. 2024TopViewRS: Vision-Language Models as Top-View Spatial Reasonerscited by 49
  21. 2024DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLMcited by 46
  22. 2024A Closer Look at the Robustness of Contrastive Language-Image Pre-Training (CLIP)cited by 44
  23. 2024Tables as Texts or Images: Evaluating the Table Reasoning Ability of LLMs and MLLMscited by 40
  24. 2023Visual Instruction Tuning with Polite Flamingocited by 38
  25. 2024In Defense of Lazy Visual Grounding for Open-Vocabulary Semantic Segmentationcited by 38
  26. 2024End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answeringcited by 37
  27. 20254D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Modelscited by 35
  28. 2024List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMscited by 32
  29. 2024Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Modelscited by 30
  30. 2024CityLLaVA: Efficient Fine-Tuning for VLMs in City Scenariocited by 27
  31. 2024Learning Visual Grounding from Generative Vision and Language Modelcited by 27
  32. 2024CoLLaVO: Crayon Large Language and Vision mOdelcited by 26
  33. 2025Introducing Visual Perception Token into Multimodal Large Language Modelcited by 26
  34. 2024Test-Time Adaptation with SaLIP: A Cascade of SAM and CLIP for Zero shot Medical Image Segmentationcited by 25
  35. 2024Artemis: Towards Referential Understanding in Complex Videoscited by 25
  36. 2024RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognitioncited by 24
  37. 2024AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Descriptioncited by 23
  38. 2024Attention Prompting on Image for Large Vision-Language Modelscited by 23

50 most cited of 218 All 218 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts
  2. 2025Helping CLIP See Both the Forest and the Trees: A Decomposition and Description Approach
  3. 2025Generalizable Prompt Learning of CLIP: A Brief Overview
  4. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
  5. 2024Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encoder
  6. 2023PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contexts
  7. 2025VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captions
  8. 2022CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigation