mlsanityData to 7 Oct 2026

LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models

arXiv 2312.02949 ↗First posted 5 Dec 2023cs.CVcited by 113

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

With the recent significant advancements in large multi-modal models (LMMs), the importance of their grounding capability in visual chat is increasingly recognized. Despite recent efforts to enable LMMs to support grounding, their capabilities for grounding and chat are usually separate, and their chat performance drops dramatically when asked to ground. The problem is the lack of a dataset for grounded visual chat (GVC). Existing grounding datasets only contain short captions. To address this issue, we have created GVC data that allows for the combination of grounding and chat capabilities. To better evaluate the GVC capabilities, we have introduced a benchmark called Grounding-Bench. Additionally, we have proposed a model design that can support GVC and various types of visual prompts by connecting segmentation models with language models. Experimental results demonstrate that our model outperforms other LMMs on Grounding-Bench. Furthermore, our model achieves competitive performance on classic grounding benchmarks like RefCOCO/+/g and Flickr30K Entities. Our code will be released at https://github.com/UX-Decoder/LLaVA-Grounding .

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 48 references that are in the map. First posted on arXiv 5 Dec 2023.

Builds on

38 of 48 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  2. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  4. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling23 Nov

Other references · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  11. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  12. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
22 more
  1. 2021Grounded Language-Image Pre-trainingcited by 1,147
  2. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  3. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  4. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  5. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  6. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  7. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  8. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  9. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  10. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  11. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  12. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 197
  13. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
  14. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  15. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 167
  16. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 147
  17. 2022SeqTR: A Simple yet Universal Network for Visual Groundingcited by 138
  18. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  19. 2023DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignmentcited by 90
  20. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Modelscited by 59
  21. 2021Towards Language-guided Visual Recognition via Dynamic Convolutionscited by 17
  22. 2022A Unified Mutual Supervision Framework for Referring Expression Segmentation and Generationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 48; 2025: 43; 2026: 22.

113 papers.

Sort:

By area:

  1. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  2. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  3. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  4. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  5. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  6. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  7. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 123
  8. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 118
  9. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  10. 2024SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Modelcited by 107
  11. 2024GroundingGPT:Language Enhanced Multi-modal Grounding Modelcited by 101
  12. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
38 more
  1. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 96
  2. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 89
  3. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  4. 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMscited by 73
  5. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMscited by 62
  6. 2024Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Modelscited by 52
  7. 2024Toward Visual Grounding: A Surveycited by 52
  8. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Promptscited by 47
  9. 2024Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgerycited by 39
  10. 2025VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generationcited by 39
  11. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 38
  12. 2024LLaVA-KD: A Framework of Distilling Multimodal Large Language Modelscited by 37
  13. 2024Grounding Multimodal Large Language Models in Actionscited by 35
  14. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  15. 2024SegPoint: Segment Any Point Cloud via Large Language Modelcited by 34
  16. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Modelscited by 33
  17. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  18. 2024Language-Image Models with 3D Understandingcited by 31
  19. 2024ChatRex: Taming Multimodal LLM for Joint Perception and Understandingcited by 30
  20. 2025SpaceVLLM: Endowing Multimodal Large Language Model with Spatio-Temporal Video Grounding Capabilitycited by 28
  21. 2024F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactionscited by 24
  22. 2024MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuningcited by 22
  23. 2025SciVerse: Unveiling the Knowledge Comprehension and Visual Reasoning of LMMs on Multi-modal Scientific Problemscited by 22
  24. 2024FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehensioncited by 20
  25. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 20
  26. 2024Training-Free Open-Ended Object Detection and Segmentation via Attention as Promptscited by 15
  27. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  28. 2024VMAD: Visual-enhanced Multimodal Large Language Model for Zero-Shot Anomaly Detectioncited by 13
  29. 2024UnSeg: One Universal Unlearnable Example Generator is Enough against All Image Segmentationcited by 13
  30. 2025LeapVAD: A Leap in Autonomous Driving via Cognitive Perception and Dual-Process Thinkingcited by 13
  31. 2025Region-based Cluster Discrimination for Visual Representation Learningcited by 13
  32. 2025ProAPO: Progressively Automatic Prompt Optimization for Visual Classificationcited by 12
  33. 2025STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Groundingcited by 11
  34. 2025Visual Intention Grounding for Egocentric Assistantscited by 11
  35. 2025Fine-grained Spatiotemporal Grounding on Egocentric Videoscited by 11
  36. 2024EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Datacited by 10
  37. 2024Advancing Fine-Grained Visual Understanding with Multi-Scale Alignment in Multi-Modal Modelscited by 10
  38. 2024DOGR: Towards Versatile Visual Document Grounding and Referringcited by 10

50 most cited of 113 All 113 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023PG-Video-LLaVA: Pixel Grounding Large Video-Language Models
  2. 2023GLaMM: Pixel Grounding Large Multimodal Model
  3. 2024Emergent Visual Grounding in Large Multimodal Models Without Grounding Supervision
  4. 2025ExpVG: Investigating the Design Space of Visual Grounding in Multimodal Large Language Model
  5. 2024F-LMM: Grounding Frozen Large Multimodal Models
  6. 2026EGM: Efficient Visual Grounding Language Models
  7. 2023Evaluation and Enhancement of Semantic Grounding in Large Vision-Language Models
  8. 2023NExT-Chat: An LMM for Chat, Detection and Segmentation