mlsanityData to 14 Oct 2026

Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V

arXiv 2310.11441 ↗First posted 17 Oct 2023cs.CV cs.AI cs.CLcited by 299

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We present Set-of-Mark (SoM), a new visual prompting method, to unleash the visual grounding abilities of large multimodal models (LMMs), such as GPT-4V. As illustrated in Fig. 1 (right), we employ off-the-shelf interactive segmentation models, such as SEEM/SAM, to partition an image into regions at different levels of granularity, and overlay these regions with a set of marks e.g., alphanumerics, masks, boxes. Using the marked image as input, GPT-4V can answer the questions that require visual grounding. We perform a comprehensive empirical study to validate the effectiveness of SoM on a wide range of fine-grained vision and multimodal tasks. For example, our experiments show that GPT-4V with SoM in zero-shot setting outperforms the state-of-the-art fully-finetuned referring expression comprehension and segmentation model on RefCOCOg. Code for SoM prompting is made public at: https://github.com/microsoft/SoM.

Key facts. Cited by 299 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 48 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

34 of 48 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct

Other references · 33

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  9. 2021Grounded Language-Image Pre-trainingcited by 1,156
  10. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  11. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  12. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
21 more
  1. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  2. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  3. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  4. 2023Segment Everything Everywhere All at Oncecited by 548
  5. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  6. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  7. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  8. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 344
  9. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 318
  10. 2022Visual Prompting via Image Inpaintingcited by 250
  11. 2023Personalize Segment Anything Model with One Shotcited by 250
  12. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  13. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  14. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
  15. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 177
  16. 2022DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detectioncited by 176
  17. 2022FocalClick: Towards Practical Interactive Image Segmentationcited by 102
  18. 2023Can SAM Segment Anything? When SAM Meets Camouflaged Object Detectioncited by 100
  19. 2023Segment Anything Meets Point Trackingcited by 81
  20. 2023SAMAug: Point Prompt Augmentation for Segment Anything Modelcited by 38
  21. 2022Open-Vocabulary 3D Detection via Image-level Class and Debiased Cross-modal Contrastive Learningcited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 83; 2025: 151; 2026: 58.

299 papers.

Sort:

By area:

  1. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  2. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
  3. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  4. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 363
  5. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 326
  6. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 311
  7. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Taskscited by 232
  8. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 221
  9. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  10. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 190
  11. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 174
  12. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
38 more
  1. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 161
  2. 2024The BrowserGym Ecosystem for Web Agent Researchcited by 161
  3. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 157
  4. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 154
  5. 2024Tree Search for Language Model Agentscited by 151
  6. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 150
  7. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 147
  8. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 140
  9. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 139
  10. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 131
  11. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 125
  12. 2025GRIT: Teaching MLLMs to Think with Imagescited by 125
  13. 2024Attacking Vision-Language Computer Agents via Pop-upscited by 116
  14. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 114
  15. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 113
  16. 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 111
  17. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 109
  18. 2025SafeArena: Evaluating the Safety of Autonomous Web Agentscited by 102
  19. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 101
  20. 2024LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Modelscited by 96
  21. 2024WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Taskscited by 90
  22. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  23. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  24. 2025ReFocus: Visual Editing as a Chain of Thought for Structured Image Understandingcited by 83
  25. 2025AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agentscited by 82
  26. 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluationcited by 81
  27. 2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuningcited by 79
  28. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 77
  29. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 76
  30. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 76
  31. 2024AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agentscited by 74
  32. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 72
  33. 2024Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Charactercited by 68
  34. 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65
  35. 2024Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Trainingcited by 64
  36. 2025A Survey on (M)LLM-Based GUI Agentscited by 62
  37. 2023EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Modelscited by 60
  38. 2025GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensingcited by 57

50 most cited of 299 All 299 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Toward Visual Grounding: A Survey
  2. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)
  3. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models
  4. 2023Fine-Grained Visual Prompting
  5. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs
  6. 2026GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models
  7. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning
  8. 2023VRPTEST: Evaluating Visual Referring Prompting in Large Multimodal Models