mlsanityData to 7 Oct 2026

Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want

arXiv 2403.20271 ↗First posted 29 Mar 2024cs.CVcited by 98

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

In this paper, we present the Draw-and-Understand framework, exploring how to integrate visual prompting understanding capabilities into Multimodal Large Language Models (MLLMs). Visual prompts allow users to interact through multi-modal instructions, enhancing the models' interactivity and fine-grained image comprehension. In this framework, we propose a general architecture adaptable to different pre-trained MLLMs, enabling it to recognize various types of visual prompts (such as points, bounding boxes, and free-form shapes) alongside language understanding. Additionally, we introduce MDVP-Instruct-Data, a multi-domain dataset featuring 1.2 million image-visual prompt-text triplets, including natural images, document images, scene text images, mobile/web screenshots, and remote sensing images. Building on this dataset, we introduce MDVP-Bench, a challenging benchmark designed to evaluate a model's ability to understand visual prompting instructions. The experimental results demonstrate that our framework can be easily and effectively applied to various MLLMs, such as SPHINX-X and LLaVA. After training with MDVP-Instruct-Data and image-level instruction datasets, our models exhibit impressive multimodal interaction capabilities and pixel-level understanding, while maintaining their image-level visual perception performance.

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 88 of its 112 references that are in the map. First posted on arXiv 29 Mar 2024.

Builds on

88 of 112 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models8 Feb
  2. 2023Osprey: Pixel Understanding with Visual Instruction Tuning15 Dec
  3. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts1 Dec
  4. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  5. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  6. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V17 Oct
  7. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  8. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models25 Aug
  9. 2023RegionBLIP: A Unified Multi-modal Pre-training Framework for Holistic and Regional Comprehension3 Aug
  10. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning18 Jul
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  12. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
1 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 75

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Segment Anythingcited by 8,158
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022A ConvNet for the 2020scited by 3,545
38 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  7. 2019Towards VQA Models That Can Readcited by 1,696
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  9. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  11. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  12. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  13. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  14. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  15. 2016Modeling Context in Referring Expressionscited by 1,054
  16. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  17. 2017DOTA: A Large-scale Dataset for Object Detection in Aerial Imagescited by 622
  18. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  19. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  20. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  21. 2023Segment Everything Everywhere All at Oncecited by 546
  22. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  23. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  24. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  25. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  26. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  27. 2019PubLayNet: largest dataset ever for document layout analysiscited by 349
  28. 2019Object Detection in Optical Remote Sensing Images: A Survey and A New Benchmarkcited by 339
  29. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  30. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  31. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensingcited by 320
  32. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  33. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  34. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  35. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  36. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 224
  37. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  38. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 60; 2026: 20.

98 papers.

Sort:

By area:

  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  2. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  3. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  4. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  5. 2025WoW: Towards a World omniscient World model Through Embodied Interactioncited by 66
  6. 2024Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Modelscited by 64
  7. 2025TIIF-Bench: How Does Your T2I Model Follow Your Instructions?cited by 64
  8. 2024MC-LLaVA: Multi-Concept Personalized Vision-Language Modelcited by 53
  9. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 52
  10. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 49
  11. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  12. 2025Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMscited by 41
38 more
  1. 2025EnerVerse-AC: Envisioning Embodied Environments with Action Conditioncited by 40
  2. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  3. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Modelscited by 33
  4. 2024ChatRex: Taming Multimodal LLM for Joint Perception and Understandingcited by 30
  5. 2024EarthMarker: A Visual Prompting Multi-modal Large Language Model for Remote Sensingcited by 29
  6. 2024CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Modelscited by 29
  7. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  8. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  9. 2025MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoderscited by 25
  10. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 24
  11. 2025VERIFY: A Benchmark of Visual Explanation and Reasoning for Investigating Multimodal Reasoning Fidelitycited by 23
  12. 2025Pixel-SAIL: Single Transformer For Pixel-Grounded Understandingcited by 22
  13. 2025VABench: A Comprehensive Benchmark for Audio-Video Generationcited by 22
  14. 2025Adaptive Classifier-Free Guidance via Dynamic Low-Confidence Maskingcited by 21
  15. 2025Prompt-based Adaptation in Large-scale Vision Models: A Surveycited by 21
  16. 2025PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularitycited by 21
  17. 2024AIC MLLM: Autonomous Interactive Correction MLLM for Robust Robotic Manipulationcited by 20
  18. 2025EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?cited by 20
  19. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 20
  20. 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenescited by 19
  21. 2025SCALAR: Scale-wise Controllable Visual Autoregressive Learningcited by 18
  22. 2025Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMscited by 18
  23. 2025Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMscited by 16
  24. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real Worldcited by 16
  25. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  26. 2025LoVR: A Benchmark for Long Video Retrieval in Multimodal Contextscited by 14
  27. 2025MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal outputcited by 14
  28. 2025Rethinking Driving World Model as Synthetic Data Generator for Perception Taskscited by 14
  29. 2025VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMscited by 12
  30. 2025EarthGPT-X: A Spatial MLLM for Multi-level Multi-Source Remote Sensing Imagery Understanding with Visual Promptingcited by 11
  31. 2025CrossLMM: Decoupling Long Video Sequences from LMMs via Dual Cross-Attention Mechanismscited by 11
  32. 2025RynnEC: Bringing MLLMs into Embodied Worldcited by 11
  33. 2025Text2SQL-Flow: A Robust SQL-Aware Data Augmentation Framework for Text-to-SQLcited by 11
  34. 2024DOGR: Towards Versatile Visual Document Grounding and Referringcited by 10
  35. 2025V2P-Bench: Evaluating Video-Language Understanding with Visual Prompts for Better Human-Model Interactioncited by 10
  36. 2025NeSyGeo: A Neuro-Symbolic Framework for Multimodal Geometric Reasoning Data Generationcited by 10
  37. 2025Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMscited by 10
  38. 2026SAMTok: Representing Any Mask with Two Wordscited by 10

50 most cited of 98 All 98 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring the Transferability of Visual Prompting for Multimodal Large Language Models
  2. 2025VP-Bench: A Comprehensive Benchmark for Visual Prompting in Multimodal Large Language Models
  3. 2024Rethinking Visual Prompting for Multimodal Large Language Models with External Knowledge
  4. 2025QG-CoC: Question-Guided Chain-of-Captions for Large Multimodal Models
  5. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge
  6. 2024EAGLE: Towards Efficient Arbitrary Referring Visual Prompts Comprehension for Multimodal Large Language Models
  7. 2025VisualQuest: A Benchmark for Abstract Visual Reasoning in MLLMs
  8. 2025VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning