mlsanityData to 7 Oct 2026

Visual Position Prompt for MLLM based Visual Grounding

arXiv 2503.15426 ↗First posted 19 Mar 2025cs.CV cs.AIcited by 16

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Although Multimodal Large Language Models (MLLMs) excel at various image-related tasks, they encounter challenges in precisely aligning coordinates with spatial information within images, particularly in position-aware tasks such as visual grounding. This limitation arises from two key factors. First, MLLMs lack explicit spatial references, making it difficult to associate textual descriptions with precise image locations. Second, their feature extraction processes prioritize global context over fine-grained spatial details, leading to weak localization capability. To address these issues, we introduce VPP-LLaVA, an MLLM enhanced with Visual Position Prompt (VPP) to improve its grounding capability. VPP-LLaVA integrates two complementary mechanisms: the global VPP overlays a learnable, axis-like tensor onto the input image to provide structured spatial cues, while the local VPP incorporates position-aware queries to support fine-grained localization.To effectively train our model with spatial guidance, we further introduce VPP-SFT, a curated dataset of 0.6M high-quality visual grounding samples. Designed in a compact format, it enables efficient training and is significantly smaller than datasets used by other MLLMs (e.g., ~21M samples in MiniGPT-v2), yet still provides a strong performance boost. The resulting model, VPP-LLaVA, not only achieves state-of-the-art results on standard visual grounding benchmarks but also demonstrates strong zero-shot generalization to challenging unseen datasets. The code and dataset are available at https://github.com/WayneTomas/VPP-LLaVA.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 66 of its 86 references that are in the map. First posted on arXiv 19 Mar 2025.

Builds on

66 of 86 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models19 Apr
  2. 2024Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Visual-Language Co-Referring14 Mar
  3. 2024GroundingGPT:Language Enhanced Multi-modal Grounding Model11 Jan
  4. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
  5. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  6. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  7. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  8. 2022PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models23 May

Other references · 58

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Segment Anythingcited by 8,158
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2025Qwen2.5-VL Technical Reportcited by 5,418
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2022Visual Prompt Tuningcited by 1,674
  10. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  11. 2021Scaling Vision Transformerscited by 993
  12. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
38 more
  1. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  2. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  3. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  4. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  5. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  6. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  7. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  8. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  9. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 498
  10. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  11. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  12. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  13. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  14. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 194
  15. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
  16. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 182
  17. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
  18. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  19. 2022RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Datacited by 121
  20. 2020f-BRS: Rethinking Backpropagating Refinement for Interactive Segmentationcited by 117
  21. 2023Fine-Grained Visual Promptingcited by 81
  22. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledgecited by 80
  23. 2024VRP-SAM: SAM with Visual Reference Promptcited by 75
  24. 2022Improving Visual Grounding with Visual-Linguistic Verification and Iterative Reasoningcited by 69
  25. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 56
  26. 2022Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuningcited by 50
  27. 2023Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMscited by 47
  28. 2024DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLMcited by 45
  29. 2021CTNet: Context-based Tandem Network for Semantic Segmentationcited by 43
  30. 2023Exploring Effective Factors for Improving Visual In-Context Learningcited by 43
  31. 2023CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Groundingcited by 35
  32. 2023LMEye: An Interactive Perception Network for Large Language Modelscited by 27
  33. 2023Context Disentangling and Prototype Inheriting for Robust Visual Groundingcited by 23
  34. 2022Entity-enhanced Adaptive Reconstruction Network for Weakly Supervised Referring Expression Groundingcited by 22
  35. 2024Exploring the Transferability of Visual Prompting for Multimodal Large Language Modelscited by 15
  36. 2022Fine-grained Visual-Text Prompt-Driven Self-Training for Open-Vocabulary Object Detectioncited by 14
  37. 2024Rethinking Visual Prompting for Multimodal Large Language Models with External Knowledgecited by 13
  38. 2024Multi-modal Attribute Prompting for Vision-Language Modelscited by 12

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 10.

16 papers.

Sort:

By area:

  1. 2025FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancingcited by 6
  2. 2026Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMscited by 4
  3. 2025MapTrace: Scalable Data Generation for Route Tracing on Mapscited by 3
  4. 2026TangramPuzzle: Evaluating Multimodal Large Language Models with Compositional Spatial Reasoningcited by 3
  5. 2025Training-Free Multi-View Extension of IC-Light for Textual Position-Aware Scene Relightingcited by 2
  6. 2026SSP-SAM: SAM with Semantic-Spatial Prompt for Referring Expression Segmentationcited by 2
  7. 2026MedSPOT: A Workflow-Aware Sequential Grounding Benchmark for Clinical GUIcited by 2
  8. 2025MITS: A Large-Scale Multimodal Benchmark Dataset for Intelligent Traffic Surveillancecited by 1
  9. 2025MARS2 2025 Challenge on Multimodal Reasoning: Datasets, Methods, Results, Discussion, and Outlookcited by 1
  10. 2026FUSAR-GPT : A Spatiotemporal Feature-Embedded and Two-Stage Decoupled Visual Language Model for SAR Imagerycited by 1
  11. 2025Enhancing Geo-localization for Crowdsourced Flood Imagery via LLM-Guided Attentioncited by 0
  12. 2026Explicit Uncertainty Modeling for Active CLIP Adaptation with Dual Prompt Tuningcited by 0
4 more
  1. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  2. 2026Group Preference Collapse in Personalized Multimodal Large Language Modelscited by 0
  3. 2026DeCo: Efficient Decouple-to-Couple Learning for Multi-Task Visual Groundingcited by 0
  4. 2026EviRover: Reinforcing Agentic Perception Beyond a Glancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding
  2. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs
  3. 2022Position-guided Text Prompt for Vision-Language Pre-training
  4. 2026IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models
  5. 2025C2F-Space: Coarse-to-Fine Space Grounding for Spatial Instructions using Vision-Language Models
  6. 2025ExpVG: Investigating the Design Space of Visual Grounding in Multimodal Large Language Model
  7. 2026Enhancing Part-Level Point Grounding for Any Open-Source MLLMs
  8. 2025Towards Understanding Visual Grounding in Visual Language Models