mlsanityData to 7 Oct 2026

Visual Prompting in Multimodal Large Language Models: A Survey

arXiv 2409.15310 ↗First posted 5 Sep 2024cs.LG cs.CVcited by 52

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal large language models (MLLMs) equip pre-trained large-language models (LLMs) with visual capabilities. While textual prompting in LLMs has been widely studied, visual prompting has emerged for more fine-grained and free-form visual instructions. This paper presents the first comprehensive survey on visual prompting methods in MLLMs, focusing on visual prompting, prompt generation, compositional reasoning, and prompt learning. We categorize existing visual prompts and discuss generative methods for automatic prompt annotations on the images. We also examine visual prompting methods that enable better alignment between visual encoders and backbone LLMs, concerning MLLM's visual grounding, object referring, and compositional reasoning abilities. In addition, we provide a summary of model training and in-context learning methods to improve MLLM's perception and understanding of visual prompts. This paper examines visual prompting methods developed in MLLMs and provides a vision of the future of these methods.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 73 of its 97 references that are in the map. First posted on arXiv 5 Sep 2024.

Builds on

73 of 97 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024Rethinking Visual Prompting for Multimodal Large Language Models with External Knowledge5 Jul
  2. 2024MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs24 Jun
  3. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks12 Jun
  4. 2024AIM: Let Any Multi-modal Large Language Models Embrace Efficient In-Context Learning11 Jun
  5. 2024Enhancing Multimodal Large Language Models with Multi-instance Visual Prompt Generator for Visual Representation Enrichment5 Jun
  6. 2024Auto-Encoding Morph-Tokens for Multimodal LLM3 May
  7. 2024List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs25 Apr
  8. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understanding15 Apr
  9. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want29 Mar
  10. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception5 Mar
  11. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Models19 Feb
  12. 2024CoLLaVO: Crayon Large Language and Vision mOdel17 Feb
4 more
  1. 2023VRPTEST: Evaluating Visual Referring Prompting in Large Multimodal Models7 Dec
  2. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts1 Dec
  3. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  4. 2023VPGTrans: Transfer Visual Prompt Generator across LLMs2 May

Other references · 57

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Segment Anythingcited by 8,158
  4. 2015Fast R-CNNcited by 2,930
  5. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  7. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  8. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  9. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  10. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
  11. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  12. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 197
38 more
  1. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 196
  2. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  3. 2022Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wildcited by 159
  4. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149
  5. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  6. 2023Explicit Visual Prompting for Low-Level Structure Segmentationscited by 113
  7. 2023In-Context Learning Unlocked for Diffusion Modelscited by 103
  8. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 103
  9. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 100
  10. 2022Understanding and Improving Visual Prompting: A Label-Mapping Perspectivecited by 86
  11. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 82
  12. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  13. 2023General Object Foundation Model for Images and Videos at Scalecited by 71
  14. 2023BlackVIP: Black-Box Visual Prompting for Robust Transfer Learningcited by 70
  15. 2024Safety of Multimodal Large Language Models on Images and Textscited by 70
  16. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
  17. 2024Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Trainingcited by 62
  18. 2024PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinementcited by 61
  19. 2024Affordances-Oriented Planning using Foundation Models for Continuous Vision-Language Navigationcited by 60
  20. 2023ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulationcited by 59
  21. 2024MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Controlcited by 52
  22. 2023Multimodal Prompt Perceiver: Empower Adaptiveness, Generalizability and Fidelity for All-in-One Image Restorationcited by 50
  23. 2023A Practical Survey on Zero-shot Prompt Design for In-context Learningcited by 49
  24. 2023A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertisecited by 49
  25. 2023A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineeringcited by 45
  26. 2024DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLMcited by 45
  27. 2024Arondight: Red Teaming Large Vision Language Models with Auto-generated Multi-modal Jailbreak Promptscited by 39
  28. 2024A3VLM: Actionable Articulation-Aware Vision Language Modelcited by 33
  29. 20243DMIT: 3D Multi-modal Instruction Tuning for Scene Understandingcited by 32
  30. 2024Language-Image Models with 3D Understandingcited by 31
  31. 2024Agent3D-Zero: An Agent for Zero-shot 3D Understandingcited by 29
  32. 2024CityLLaVA: Efficient Fine-Tuning for VLMs in City Scenariocited by 27
  33. 2024VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generationcited by 25
  34. 2024CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generationcited by 21
  35. 2024AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Descriptioncited by 21
  36. 2024Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuningcited by 20
  37. 2024Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Groundingcited by 18
  38. 2023InstructDET: Diversifying Referring Object Detection with Generalized Instructionscited by 16

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 24; 2026: 23.

52 papers.

Sort:

By area:

  1. 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65
  2. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  3. 2025Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulationcited by 41
  4. 2025MLLMs are Deeply Affected by Modality Biascited by 38
  5. 2025A Survey on Video Temporal Grounding with Multimodal Large Language Modelcited by 19
  6. 2025PixelVLA: Advancing Pixel-level Understanding in Vision-Language-Action Modelcited by 16
  7. 2024Vector-ICL: In-context Learning with Continuous Vector Representationscited by 13
  8. 2025Towards Online Multi-Modal Social Interaction Understandingcited by 12
  9. 2026How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editingcited by 12
  10. 2025EarthGPT-X: A Spatial MLLM for Multi-level Multi-Source Remote Sensing Imagery Understanding with Visual Promptingcited by 11
  11. 2025Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMscited by 10
  12. 2025WildScore: Benchmarking MLLMs in-the-Wild Symbolic Music Reasoningcited by 10
38 more
  1. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 8
  2. 2024MAIRA-Seg: Enhancing Radiology Report Generation with Segmentation-Aware Multimodal Large Language Modelscited by 7
  3. 2025Video Panels for Long Video Understandingcited by 6
  4. 2024Autonomous Imagination: Closed-Loop Decomposition of Visual-to-Textual Conversion in Visual Reasoning for Multimodal Large Language Modelscited by 5
  5. 2025DINO-R1: Incentivizing Reasoning Capability in Vision Foundation Modelscited by 5
  6. 2026OpenVoxel: Training-Free Grouping and Captioning Voxels for Open-Vocabulary 3D Scene Understandingcited by 5
  7. 2026SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generationcited by 5
  8. 2025RESAnything: Attribute Prompting for Arbitrary Referring Segmentationcited by 4
  9. 2025Uncertainty-o: One Model-agnostic Framework for Unveiling Uncertainty in Large Multimodal Modelscited by 4
  10. 2026AMPS: Adaptive Modality Preference Steering via Functional Entropycited by 4
  11. 2025Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instructioncited by 3
  12. 2025Abductive Logical Rule Induction by Bridging Inductive Logic Programming and Multimodal Large Language Modelscited by 3
  13. 2025RxnCaption: Reformulating Reaction Diagram Parsing as Visual Prompt Guided Captioningcited by 3
  14. 2026VisionCoach: Reinforcing Grounded Video Reasoning via Visual-Perception Promptingcited by 3
  15. 2025Plane Geometry Problem Solving with Multi-modal Reasoning: A Surveycited by 2
  16. 2025AutoV: Loss-Oriented Ranking for Visual Prompt Retrieval in LVLMscited by 2
  17. 2025PDB-Eval: An Evaluation of Large Multimodal Models for Description and Explanation of Personalized Driving Behaviorcited by 2
  18. 2025Enhancing Vision-Language Models for Autonomous Driving through Task-Specific Prompting and Spatial Reasoningcited by 2
  19. 2026ViSA-Enhanced Aerial VLN: A Visual-Spatial Reasoning Enhanced Framework for Aerial Vision-Language Navigationcited by 2
  20. 2026Leveraging Gaze and Set-of-Mark in VLLMs for Human-Object Interaction Anticipation from Egocentric Videoscited by 2
  21. 2026OLIVIA: Online Learning via Inference-time Action Adaptation for Decision Making in LLM ReAct Agentscited by 2
  22. 2025Invisible Watermarks, Visible Gains: Steering Machine Unlearning with Bi-Level Watermarking Designcited by 1
  23. 2026Chatting with Images for Introspective Visual Thinkingcited by 1
  24. 2026Do You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answeringcited by 1
  25. 2026Skill-CMIB: Multimodal Agent Skill for Consistent Action via Conditional Multimodal Information Bottleneckcited by 1
  26. 2026Who Says What: Symbolic Trimodal Binding Mechanisms in Audio-Visual LLMscited by 1
  27. 2025The Role of Background Information in Reducing Object Hallucination in Vision-Language Models: Insights from Cutoff API Promptingcited by 0
  28. 2025Finding Needles in Images: Can Multimodal LLMs Locate Fine Details?cited by 0
  29. 2025Labels or Input? Rethinking Augmentation in Multimodal Hate Detectioncited by 0
  30. 2025What Shape Is Optimal for Masks in Text Removal?cited by 0
  31. 2026Global Commander and Local Operative: A Dual-Agent Framework for Scene Navigationcited by 0
  32. 2026Visual Prompt Discovery via Semantic Explorationcited by 0
  33. 2026PhysQuantAgent: An Inference Pipeline of Mass Estimation for Vision-Language Modelscited by 0
  34. 2026Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMscited by 0
  35. 2026Rule-VLN: Bridging Perception and Compliance via Semantic Reasoning and Geometric Rectificationcited by 0
  36. 2026SketchVLM: Vision language models can annotate images to explain thoughts and guide userscited by 0
  37. 2026Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understandingcited by 0
  38. 2026[CLS] is Not Enough: Multi-Label Recognition via Patch-Level Inference and Adaptive Aggregationcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring the Transferability of Visual Prompting for Multimodal Large Language Models
  2. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models
  3. 2025VP-Bench: A Comprehensive Benchmark for Visual Prompting in Multimodal Large Language Models
  4. 2024The Revolution of Multimodal Large Language Models: A Survey
  5. 2025QG-CoC: Question-Guided Chain-of-Captions for Large Multimodal Models
  6. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge
  7. 2024Attention Prompting on Image for Large Vision-Language Models
  8. 2023Text as Images: Can Multimodal Large Language Models Follow Printed Instructions in Pixels?