mlsanityData to 7 Oct 2026

ExpVG: Investigating the Design Space of Visual Grounding in Multimodal Large Language Model

arXiv 2508.08066 ↗First posted 11 Aug 2025cs.CV cs.AI cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Fine-grained multimodal capability in Multimodal Large Language Models (MLLMs) has emerged as a critical research direction, particularly for tackling the visual grounding (VG) problem. Despite the strong performance achieved by existing approaches, they often employ disparate design choices when fine-tuning MLLMs for VG, lacking systematic verification to support these designs. To bridge this gap, this paper presents a comprehensive study of various design choices that impact the VG performance of MLLMs. We conduct our analysis using LLaVA-1.5, which has been widely adopted in prior empirical studies of MLLMs. While more recent models exist, we follow this convention to ensure our findings remain broadly applicable and extendable to other architectures. We cover two key aspects: (1) exploring different visual grounding paradigms in MLLMs, identifying the most effective design, and providing our insights; and (2) conducting ablation studies on the design of grounding data to optimize MLLMs' fine-tuning for the VG task. Finally, our findings contribute to a stronger MLLM for VG, achieving improvements of +5.6% / +6.9% / +7.0% on RefCOCO/+/g over the LLaVA-1.5.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 56 references that are in the map. First posted on arXiv 11 Aug 2025.

Builds on

50 of 56 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024What matters when building vision-language models?3 May
  2. 2023PixelLM: Pixel Reasoning with Large Multimodal Model4 Dec
  3. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
  4. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  5. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  10. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  11. 2021Grounded Language-Image Pre-trainingcited by 1,147
  12. 2016Modeling Context in Referring Expressionscited by 1,054
32 more
  1. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  2. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  3. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  4. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  5. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  6. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  7. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  8. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  9. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  10. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  11. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  12. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  13. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  14. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  15. 2016Image Captioning with Semantic Attentioncited by 260
  16. 2020Improving One-stage Visual Grounding by Recursive Sub-query Constructioncited by 137
  17. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  18. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  19. 2020Say As You Wish: Fine-grained Control of Image Caption Generation with Abstract Scene Graphscited by 87
  20. 2020On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answeringcited by 78
  21. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  22. 2022Improving Visual Grounding with Visual-Linguistic Verification and Iterative Reasoningcited by 69
  23. 2021SimCC: a Simple Coordinate Classification Perspective for Human Pose Estimationcited by 68
  24. 2017VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentationcited by 50
  25. 2023Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMscited by 47
  26. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perceptioncited by 39
  27. 2024Robin3D: Improving 3D Large Language Model via Robust Instruction Tuningcited by 26
  28. 2024Intent3D: 3D Object Detection in RGB-D Scans Based on Human Intentioncited by 16
  29. 2024SegVG: Transferring Object Bounding Box to Segmentation for Visual Groundingcited by 14
  30. 2024Infant Agent: A Tool-Integrated, Logic-Driven Agent with Cost-Effective API Usagecited by 11
  31. 2024Visual Grounding with Attention-Driven Constraint Balancingcited by 7
  32. 2024ACTRESS: Active Retraining for Semi-supervised Visual Groundingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025VGent: Visual Grounding via Modular Design for Disentangling Reasoning and Predictioncited by 6
  2. 2026Dense Coordinate-List Fine-Tuning Induces a Controllable Interference Surface in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs
  2. 2025Towards Understanding Visual Grounding in Visual Language Models
  3. 2025A Survey on Video Temporal Grounding with Multimodal Large Language Model
  4. 2024The Revolution of Multimodal Large Language Models: A Survey
  5. 2026GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models
  6. 2025Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models
  7. 2026PlaM: Training-Free Plateau-Guided Model Merging for Better Visual Grounding in MLLMs
  8. 2025GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation