mlsanityData to 7 Oct 2026

GroundingGPT:Language Enhanced Multi-modal Grounding Model

arXiv 2401.06071 ↗First posted 11 Jan 2024cs.CV cs.CLcited by 101

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Multi-modal large language models have demonstrated impressive performance across various tasks in different modalities. However, existing multi-modal models primarily emphasize capturing global information within each modality while neglecting the importance of perceiving local information across modalities. Consequently, these models lack the ability to effectively understand the fine-grained details of input data, limiting their performance in tasks that require a more nuanced understanding. To address this limitation, there is a compelling need to develop models that enable fine-grained understanding across multiple modalities, thereby enhancing their applicability to a wide range of tasks. In this paper, we propose GroundingGPT, a language enhanced multi-modal grounding model. Beyond capturing global information like other multi-modal models, our proposed model excels at tasks demanding a detailed understanding of local information within the input. It demonstrates precise identification and localization of specific regions in images or moments in videos. To achieve this objective, we design a diversified dataset construction pipeline, resulting in a multi-modal, multi-granularity dataset for model training. The code, dataset, and demo of our model can be found at https: //github.com/lzw-lzw/GroundingGPT.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 58 references that are in the map. First posted on arXiv 11 Jan 2024.

Builds on

37 of 58 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models5 Dec
  2. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
  3. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  4. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  5. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling23 Nov

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  7. 2019Towards VQA Models That Can Readcited by 1,696
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  9. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  11. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  12. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
20 more
  1. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  2. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  3. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  4. 2017Dense-Captioning Events in Videoscited by 789
  5. 2017Localizing Moments in Video with Natural Languagecited by 660
  6. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  7. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  8. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  9. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  10. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  11. 2019Clotho: An Audio Captioning Datasetcited by 354
  12. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  13. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  14. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 253
  15. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  16. 2021Localizing Visual Sounds the Hard Waycited by 169
  17. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127
  18. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  19. 2023Hierarchical Video-Moment Retrieval and Step-Captioningcited by 83
  20. 2023LLaSM: Large Language and Speech Modelcited by 56

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 55; 2026: 23.

101 papers.

Sort:

By area:

  1. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 99
  2. 2025UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learningcited by 83
  3. 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65
  4. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  5. 2025LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understandingcited by 57
  6. 2024Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Modelscited by 52
  7. 2024Toward Visual Grounding: A Surveycited by 52
  8. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 52
  9. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 48
  10. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 38
  11. 2024SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusioncited by 36
  12. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
38 more
  1. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AIcited by 30
  2. 2025MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversationcited by 28
  3. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  4. 2025Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Modelscited by 22
  5. 2024FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehensioncited by 20
  6. 2025A Survey on Video Temporal Grounding with Multimodal Large Language Modelcited by 19
  7. 2024ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videoscited by 18
  8. 2025Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Markscited by 17
  9. 2024ReWind: Understanding Long Videos with Instructed Learnable Memorycited by 16
  10. 2025Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Modelscited by 16
  11. 2025Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperationcited by 16
  12. 2025Visual Position Prompt for MLLM based Visual Groundingcited by 16
  13. 2024TimeRefine: Temporal Grounding with Time Refining Video LLMcited by 15
  14. 2025AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMscited by 15
  15. 2025TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMscited by 15
  16. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  17. 2025AMSbench: A Comprehensive Benchmark for Evaluating MLLM Capabilities in AMS Circuitscited by 14
  18. 2024UnifiedMLLM: Enabling Unified Representation for Multi-modal Multi-tasks With Large Language Modelcited by 13
  19. 2025Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generationcited by 13
  20. 2025TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Actioncited by 13
  21. 2025SurgVidLM: Towards Multi-grained Surgical Video Understanding with Large Language Modelcited by 13
  22. 2024Grounding-IQA: Grounding Multimodal Language Model for Image Quality Assessmentcited by 12
  23. 2025Enrich and Detect: Video Temporal Grounding with Multimodal LLMscited by 12
  24. 2025Visual Intention Grounding for Egocentric Assistantscited by 11
  25. 2025Improving the Reasoning of Multi-Image Grounding in MLLMs via Reinforcement Learningcited by 11
  26. 2024Advancing Fine-Grained Visual Understanding with Multi-Scale Alignment in Multi-Modal Modelscited by 10
  27. 2025PropVG: End-to-End Proposal-Driven Visual Grounding with Multi-Granularity Discriminationcited by 10
  28. 2024Dog-IQA: Standard-guided Zero-shot MLLM for Mix-grained Image Quality Assessmentcited by 9
  29. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 9
  30. 2025VG-Refiner: Towards Tool-Refined Referring Grounded Reasoning via Agentic Reinforcement Learningcited by 8
  31. 2025Connector-S: A Survey of Connectors in Multi-modal Large Language Modelscited by 7
  32. 2025New Dataset and Methods for Fine-Grained Compositional Referring Expression Comprehension via Specialist-MLLM Collaborationcited by 7
  33. 2025DisTime: Distribution-based Time Representation for Video Large Language Modelscited by 7
  34. 2025TAG: A Simple Yet Effective Temporal-Aware Approach for Zero-Shot Video Temporal Groundingcited by 7
  35. 2025TempR1: Improving Temporal Understanding of MLLMs via Temporal-Aware Multi-Task Reinforcement Learningcited by 7
  36. 2024Language-Driven Visual Consensus for Zero-Shot Semantic Segmentationcited by 6
  37. 2025ToG-Bench: Task-Oriented Spatio-Temporal Grounding in Egocentric Videoscited by 6
  38. 2025LLaFEA: Frame-Event Complementary Fusion for Fine-Grained Spatiotemporal Understanding in LMMscited by 5

50 most cited of 101 All 101 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models
  2. 2025ExpVG: Investigating the Design Space of Visual Grounding in Multimodal Large Language Model
  3. 2024CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Models
  4. 2023PG-Video-LLaVA: Pixel Grounding Large Video-Language Models
  5. 2026Focus When Necessary: Adaptive Routing and Collaborative Grounding for Training-Free Visual Grounding
  6. 2025SpaceVLLM: Endowing Multimodal Large Language Model with Spatio-Temporal Video Grounding Capability
  7. 2025MoDA: Modulation Adapter for Fine-Grained Visual Grounding in Instructional MLLMs
  8. 2026GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding