mlsanityData to 14 Oct 2026

ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts

arXiv 2312.00784 ↗First posted 1 Dec 2023cs.CV cs.AI cs.CLcited by 161

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

While existing large vision-language multimodal models focus on whole image understanding, there is a prominent gap in achieving region-specific comprehension. Current approaches that use textual coordinates or spatial encodings often fail to provide a user-friendly interface for visual prompting. To address this challenge, we introduce a novel multimodal model capable of decoding arbitrary visual prompts. This allows users to intuitively mark images and interact with the model using natural cues like a "red bounding box" or "pointed arrow". Our simple design directly overlays visual markers onto the RGB image, eliminating the need for complex region encodings, yet achieves state-of-the-art performance on region-understanding tasks like Visual7W, PointQA, and Visual Commonsense Reasoning benchmark. Furthermore, we present ViP-Bench, a comprehensive benchmark to assess the capability of models in understanding visual prompts across multiple dimensions, enabling future research in this domain. Code, data, and model are publicly available.

Key facts. Cited by 161 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 58 references that are in the map. First posted on arXiv 1 Dec 2023.

Builds on

46 of 58 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V17 Oct
  2. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  3. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  5. 2023RegionBLIP: A Unified Multi-modal Pre-training Framework for Holistic and Regional Comprehension3 Aug
  6. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  7. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning18 Jul
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  9. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Segment Anythingcited by 8,234
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2016Layer Normalizationcited by 5,669
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
23 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  3. 2019Towards VQA Models That Can Readcited by 1,718
  4. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  5. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  6. 2016Modeling Context in Referring Expressionscited by 1,063
  7. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  8. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  9. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  10. 2021Extract Free Dense Labels from CLIPcited by 583
  11. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  12. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  13. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  14. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  15. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  16. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  17. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  18. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  19. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  20. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  21. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 121
  22. 2016Modeling Relationships in Referential Expressions with Compositional Modular Networkscited by 106
  23. 2020Point and Ask: Incorporating Pointing into Visual Question Answeringcited by 39

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 49; 2025: 68; 2026: 44.

161 papers.

Sort:

By area:

  1. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 402
  2. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 190
  3. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
  4. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 139
  5. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 127
  6. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 101
  7. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 97
  8. 2024LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Modelscited by 96
  9. 2024Matryoshka Multimodal Modelscited by 84
  10. 2024VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLMcited by 72
  11. 2024A Touch, Vision, and Language Dataset for Multimodal Alignmentcited by 69
  12. 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65
38 more
  1. 2024Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Trainingcited by 64
  2. 2024Multi-Object Hallucination in Vision-Language Modelscited by 60
  3. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 54
  4. 2025FineVision: Open Data Is All You Needcited by 53
  5. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policycited by 52
  6. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 51
  7. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 45
  8. 2024VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Modelscited by 43
  9. 2024CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examplescited by 35
  10. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Modelscited by 33
  11. 2024Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Visual-Language Co-Referringcited by 29
  12. 2024EarthMarker: A Visual Prompting Multi-modal Large Language Model for Remote Sensingcited by 29
  13. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 28
  14. 2024CityLLaVA: Efficient Fine-Tuning for VLMs in City Scenariocited by 27
  15. 2024FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularitycited by 27
  16. 2024CoLLaVO: Crayon Large Language and Vision mOdelcited by 26
  17. 2024AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Descriptioncited by 23
  18. 2024MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation modelscited by 23
  19. 2025Pixel-SAIL: Single Transformer For Pixel-Grounded Understandingcited by 22
  20. 2025Robotic Visual Instructioncited by 22
  21. 2024R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interestcited by 21
  22. 2025EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?cited by 21
  23. 2025PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularitycited by 21
  24. 2024Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves?cited by 20
  25. 2024AIC MLLM: Autonomous Interactive Correction MLLM for Robust Robotic Manipulationcited by 20
  26. 2024Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentationcited by 20
  27. 2025Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Markscited by 17
  28. 2024PA-LLaVA: A Large Language-Vision Assistant for Human Pathology Image Understandingcited by 16
  29. 2025Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMscited by 16
  30. 2025Visual Position Prompt for MLLM based Visual Groundingcited by 16
  31. 2024One is Plenty: A Polymorphic Feature Interpreter for Immutable Heterogeneous Collaborative Perceptioncited by 15
  32. 2025MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal outputcited by 15
  33. 2025DeepSketcher: Internalizing Visual Manipulation for Multimodal Reasoningcited by 14
  34. 2024Rethinking Visual Prompting for Multimodal Large Language Models with External Knowledgecited by 13
  35. 2024PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generationcited by 13
  36. 2025Object-centric Video Question Answering with Visual Grounding and Referringcited by 13
  37. 2025Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Imagescited by 13
  38. 2025Object-centric Binding in Contrastive Language-Image Pretrainingcited by 12

50 most cited of 161 All 161 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VP-Bench: A Comprehensive Benchmark for Visual Prompting in Multimodal Large Language Models
  2. 2024Exploring the Transferability of Visual Prompting for Multimodal Large Language Models
  3. 2026UnAC: Adaptive Visual Prompting with Abstraction and Stepwise Checking for Complex Multimodal Reasoning
  4. 2023VRPTEST: Evaluating Visual Referring Prompting in Large Multimodal Models
  5. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge
  6. 2025Understand, Think, and Answer: Advancing Visual Reasoning with Large Multimodal Models
  7. 2024Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models
  8. 2025How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding