mlsanityData to 7 Oct 2026

Fine-Grained Visual Prompting

arXiv 2306.04356 ↗First posted 7 Jun 2023cs.CVcited by 81

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Vision-Language Models (VLMs), such as CLIP, have demonstrated impressive zero-shot transfer capabilities in image-level visual perception. However, these models have shown limited performance in instance-level tasks that demand precise localization and recognition. Previous works have suggested that incorporating visual prompts, such as colorful boxes or circles, can improve the ability of models to recognize objects of interest. Nonetheless, compared to language prompting, visual prompting designs are rarely explored. Existing approaches, which employ coarse visual cues such as colorful boxes or circles, often result in sub-optimal performance due to the inclusion of irrelevant and noisy pixels. In this paper, we carefully study the visual prompting designs by exploring more fine-grained markings, such as segmentation masks and their variations. In addition, we introduce a new zero-shot framework that leverages pixel-level annotations acquired from a generalist segmentation model for fine-grained visual prompting. Consequently, our investigation reveals that a straightforward application of blur outside the target mask, referred to as the Blur Reverse Mask, exhibits exceptional effectiveness. This proposed prompting strategy leverages the precise mask annotations to reduce focus on weakly related regions while retaining spatial coherence between the target and the surrounding background. Our Fine-Grained Visual Prompting (FGVP) demonstrates superior performance in zero-shot comprehension of referring expressions on the RefCOCO, RefCOCO+, and RefCOCOg benchmarks. It outperforms prior methods by an average margin of 3.0% to 4.6%, with a maximum improvement of 12.5% on the RefCOCO+ testA subset. Code is available at https://github.com/ylingfeng/FGVP.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 61 of its 69 references that are in the map. First posted on arXiv 7 Jun 2023.

Builds on

61 of 69 references.

References · 61

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2023Segment Anythingcited by 8,158
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2017Mask R-CNNcited by 7,259
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  7. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  8. 2016Pyramid Scene Parsing Networkcited by 2,472
  9. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  10. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  11. 2022Visual Prompt Tuningcited by 1,674
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  13. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  14. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  15. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  16. 2021Grounded Language-Image Pre-trainingcited by 1,147
  17. 2016Modeling Context in Referring Expressionscited by 1,054
  18. 2019When Does Label Smoothing Help?cited by 985
  19. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  20. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  21. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  22. 2018Panoptic Segmentationcited by 796
  23. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  24. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  25. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  26. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  27. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  28. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  29. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  30. 2019SOLO: Segmenting Objects by Locationscited by 289
  31. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  32. 2022Visual Prompting via Image Inpaintingcited by 248
  33. 2014Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Partscited by 230
  34. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Modelcited by 223
  35. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  36. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  37. 2023SegGPT: Segmenting Everything In Contextcited by 193
  38. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 33; 2025: 33; 2026: 12.

81 papers.

Sort:

By area:

  1. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  2. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  3. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 126
  4. 2024PromptKD: Unsupervised Prompt Distillation for Vision-Language Modelscited by 72
  5. 2024DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLMcited by 45
  6. 2024SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusioncited by 36
  7. 20254D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Modelscited by 35
  8. 2024Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graphcited by 32
  9. 2024Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Modelscited by 30
  10. 2025Introducing Visual Perception Token into Multimodal Large Language Modelcited by 26
  11. 2024Test-Time Adaptation with SaLIP: A Cascade of SAM and CLIP for Zero shot Medical Image Segmentationcited by 25
  12. 2024RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognitioncited by 24
38 more
  1. 2024Attention Prompting on Image for Large Vision-Language Modelscited by 23
  2. 2025Prompt-based Adaptation in Large-scale Vision Models: A Surveycited by 21
  3. 2024Chat2Layout: Interactive 3D Furniture Layout with a Multimodal LLMcited by 20
  4. 2023InstructDET: Diversifying Referring Object Detection with Generalized Instructionscited by 16
  5. 2024Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentationcited by 16
  6. 2025Visual Position Prompt for MLLM based Visual Groundingcited by 16
  7. 2024Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmerscited by 14
  8. 2024Towards Open-World Grasping with Large Vision-Language Modelscited by 13
  9. 2025Constructive Distortion: Improving MLLMs with Attention-Guided Image Warpingcited by 13
  10. 2024Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledgecited by 12
  11. 2025Towards Online Multi-Modal Social Interaction Understandingcited by 12
  12. 2025MotionSight: Boosting Fine-Grained Motion Understanding in Multimodal LLMscited by 11
  13. 2024Right this way: Can VLMs Guide Us to See More to Answer Questions?cited by 10
  14. 2025Through the Magnifying Glass: Adaptive Perception Magnification for Hallucination-Free VLM Decodingcited by 10
  15. 2025Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptationcited by 10
  16. 2024WeakSAM: Segment Anything Meets Weakly-supervised Instance-level Recognitioncited by 9
  17. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 9
  18. 2024GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognitioncited by 8
  19. 2025Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generationcited by 8
  20. 2024FALIP: Visual Prompt as Foveal Attention Boosts CLIP Zero-Shot Performancecited by 7
  21. 2024VACoDe: Visual Augmented Contrastive Decodingcited by 7
  22. 2025Visual and Textual Prompts in VLLMs for Enhancing Emotion Recognitioncited by 7
  23. 2025TextRegion: Text-Aligned Region Tokens from Frozen Image-Text Modelscited by 7
  24. 2025Decouple before Align: Visual Disentanglement Enhances Prompt Tuningcited by 7
  25. 2024Learning Visual Prompts for Guiding the Attention of Vision Transformerscited by 6
  26. 2024Visual Prompt Engineering for Vision Language Models in Radiologycited by 6
  27. 2025Exploring Multimodal Prompt for Visualization Authoring with Large Language Modelscited by 5
  28. 2026MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptationcited by 5
  29. 2025RESAnything: Attribute Prompting for Arbitrary Referring Segmentationcited by 4
  30. 2025SAMPO:Scale-wise Autoregression with Motion PrOmpt for generative world modelscited by 4
  31. 2023Segment, Select, Correct: A Framework for Weakly-Supervised Referring Segmentationcited by 3
  32. 2024Data-Efficient 3D Visual Grounding via Order-Aware Referringcited by 3
  33. 2024Robust Adaptation of Foundation Models with Black-Box Visual Promptingcited by 3
  34. 2024CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Modelscited by 3
  35. 2025VaMP: Variational Multi-Modal Prompt Learning for Vision-Language Modelscited by 3
  36. 2026ViKey: Enhancing Temporal Understanding in Videos via Visual Promptingcited by 3
  37. 2026Unification of Closed-Open Industrial Detection Scenarios: New Large-Scale Benchmarks,Challenges and Baselinescited by 3
  38. 2024Spurious Feature Eraser: Stabilizing Test-Time Adaptation for Vision-Language Foundation Modelcited by 2

50 most cited of 81 All 81 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Visual Prompting in Multimodal Large Language Models: A Survey
  2. 2024Multi-modal Attribute Prompting for Vision-Language Models
  3. 2025STOP: Integrated Spatial-Temporal Dynamic Prompting for Video Understanding
  4. 2025Show or Tell? Effectively prompting Vision-Language Models for semantic segmentation
  5. 2024When Does Visual Prompting Outperform Linear Probing for Vision-Language Models? A Likelihood Perspective
  6. 2024Exploring the Transferability of Visual Prompting for Multimodal Large Language Models
  7. 2024Exploring the Zero-Shot Capabilities of Vision-Language Models for Improving Gaze Following
  8. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V