mlsanityData to 7 Oct 2026

GROUNDHOG: Grounding Large Language Models to Holistic Segmentation

arXiv 2402.16846 ↗First posted 26 Feb 2024cs.CV cs.AI cs.CLcited by 70

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Most multimodal large language models (MLLMs) learn language-to-object grounding through causal language modeling where grounded objects are captured by bounding boxes as sequences of location tokens. This paradigm lacks pixel-level representations that are important for fine-grained visual understanding and diagnosis. In this work, we introduce GROUNDHOG, an MLLM developed by grounding Large Language Models to holistic segmentation. GROUNDHOG incorporates a masked feature extractor and converts extracted features into visual entity tokens for the MLLM backbone, which then connects groundable phrases to unified grounding masks by retrieving and merging the entity masks. To train GROUNDHOG, we carefully curated M3G2, a grounded visual instruction tuning dataset with Multi-Modal Multi-Grained Grounding, by harvesting a collection of segmentation-grounded datasets with rich annotations. Our experimental results show that GROUNDHOG achieves superior performance on various language grounding tasks without task-specific fine-tuning, and significantly reduces object hallucination. GROUNDHOG also demonstrates better grounding towards complex forms of visual input and provides easy-to-understand diagnosis in failure cases.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 79 of its 95 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

79 of 95 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  2. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  4. 2023World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Models14 Jun
  5. 2023Contextual Object Detection with Multimodal Large Language Models29 May
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  7. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling23 Nov

Other references · 72

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Segment Anythingcited by 8,158
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  10. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  12. 2019Towards VQA Models That Can Readcited by 1,696
38 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  2. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  3. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  4. 2021Grounded Language-Image Pre-trainingcited by 1,147
  5. 2016Modeling Context in Referring Expressionscited by 1,054
  6. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  7. 2023A Survey on Multimodal Large Language Modelscited by 898
  8. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  9. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  10. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  11. 2023Vision Transformers Need Registerscited by 844
  12. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  13. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  14. 2018Panoptic Segmentationcited by 796
  15. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  16. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  17. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  18. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  19. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  20. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  21. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  22. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  23. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  24. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  25. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  26. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  27. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  28. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  29. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  30. 2019Connecting Vision and Language with Localized Narrativescited by 225
  31. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
  32. 2022Vision GNN: An Image is Worth Graph of Nodescited by 185
  33. 2021TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene textcited by 170
  34. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  35. 2020BoxInst: High-Performance Instance Segmentation with Box Annotationscited by 150
  36. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 147
  37. 2023PACO: Parts and Attributes of Common Objectscited by 146
  38. 2020PhraseCut: Language-based Image Segmentation in the Wildcited by 116

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 22; 2025: 35; 2026: 13.

70 papers.

Sort:

By area:

  1. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  2. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  3. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  4. 2024Multi-Object Hallucination in Vision-Language Modelscited by 60
  5. 2024Text4Seg: Reimagining Image Segmentation as Text Generationcited by 59
  6. 2024Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguitiescited by 58
  7. 2025MindJourney: Test-Time Scaling with World Models for Spatial Reasoningcited by 49
  8. 2024Grounded Multi-Hop VideoQA in Long-Form Egocentric Videoscited by 43
  9. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 38
  10. 20243D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucinationcited by 36
  11. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  12. 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognitioncited by 35
38 more
  1. 2024F-LMM: Grounding Frozen Large Multimodal Modelscited by 33
  2. 2024VividMed: Vision Language Model with Versatile Visual Grounding for Medicinecited by 28
  3. 2024DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiencescited by 27
  4. 2025UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interfacecited by 27
  5. 2024Attention Prompting on Image for Large Vision-Language Modelscited by 23
  6. 2025POPEN: Preference-Based Optimization and Ensemble for LVLM-Based Reasoning Segmentationcited by 22
  7. 2025Pixel-SAIL: Single Transformer For Pixel-Grounded Understandingcited by 22
  8. 2025HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Modelcited by 19
  9. 2024Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Levelcited by 18
  10. 2025MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal outputcited by 14
  11. 2024Pseudo-RIS: Distinctive Pseudo-supervision Generation for Referring Image Segmentationcited by 13
  12. 2025LISAT: Language-Instructed Segmentation Assistant for Satellite Imagerycited by 13
  13. 2025Towards Cross-View Point Correspondence in Vision-Language Modelscited by 12
  14. 2024VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentationcited by 11
  15. 2025Towards Understanding How Knowledge Evolves in Large Vision-Language Modelscited by 10
  16. 2024Multi-View and Multi-Scale Alignment for Contrastive Language-Image Pre-training in Mammographycited by 9
  17. 2024Emergent Visual Grounding in Large Multimodal Models Without Grounding Supervisioncited by 8
  18. 2025Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generationcited by 8
  19. 2025Text4Seg++: Advancing Image Segmentation via Generative Language Modelingcited by 8
  20. 2025Improving Generalized Visual Grounding with Instance-aware Joint Learningcited by 8
  21. 2026On the Generalization Capacities of MLLMs for Spatial Intelligencecited by 7
  22. 2024Scene-Graph ViT: End-to-End Open-Vocabulary Visual Relationship Detectioncited by 6
  23. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
  24. 2025ALTo: Adaptive-Length Tokenizer for Autoregressive Mask Generationcited by 5
  25. 2025Refer to Any Segmentation Mask Group With Vision-Language Promptscited by 5
  26. 2025PostAlign: Multimodal Grounding as a Corrective Lens for MLLMscited by 5
  27. 2025The Mechanistic Emergence of Symbol Grounding in Language Modelscited by 5
  28. 2025ARGenSeg: Image Segmentation with Autoregressive Image Generation Modelcited by 5
  29. 2026Towards Pixel-Level VLM Perception via Simple Points Predictioncited by 5
  30. 2024ROSE: Revolutionizing Open-Set Dense Segmentation with Patch-Wise Perceptual Large Multimodal Modelcited by 4
  31. 2025GroundCap: A Visually Grounded Image Captioning Datasetcited by 4
  32. 2025From Objects to Events: Unlocking Complex Visual Understanding in Object Detectors via LLM-guided Symbolic Reasoningcited by 3
  33. 2025StoryReasoning Dataset: Using Chain-of-Thought for Scene Understanding and Grounded Story Generationcited by 3
  34. 2025SGDiff: Scene Graph Guided Diffusion Model for Image Collaborative SegCaptioningcited by 3
  35. 2024EditScout: Locating Forged Regions from Diffusion-based Edited Images with Multimodal LLMcited by 2
  36. 2025What Changed? Detecting and Evaluating Instruction-Guided Image Edits with Multimodal Large Language Modelscited by 2
  37. 2025Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniquescited by 2
  38. 2025Grounding Everything in Tokens for Multimodal Large Language Modelscited by 2

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation
  2. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding
  3. 2026IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models
  4. 2026Reasoning-Guided Part-Level Visual Grounding via Reinforcement Learning
  5. 2025Hierarchical Contextual Grounding LVLM: Enhancing Fine-Grained Visual-Language Understanding with Robust Grounding
  6. 2025Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning
  7. 2024Learning Visual Grounding from Generative Vision and Language Model
  8. 2026Focus When Necessary: Adaptive Routing and Collaborative Grounding for Training-Free Visual Grounding