mlsanityData to 7 Oct 2026

BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs

arXiv 2307.08581 ↗First posted 17 Jul 2023cs.CV cs.AIcited by 114

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

LLMs have demonstrated remarkable abilities at interacting with humans through language, especially with the usage of instruction-following data. Recent advancements in LLMs, such as MiniGPT-4, LLaVA, and X-LLM, further enlarge their abilities by incorporating multi-modal inputs, including image, video, and speech. Despite their effectiveness at generating precise and detailed language understanding of the given modality signal, these LLMs give up the ability to ground specific parts of inputs, thus only constructing a coarse-grained mapping. However, explicit and informative correspondence between text and other modalities will not only improve the user experience but also help to expand the application scenario of multi-modal LLMs. Therefore, we propose BuboGPT, a multi-modal LLM with visual grounding that can perform cross-modal interaction between vision, audio and language, providing fine-grained understanding of visual objects and other given modalities. As a result, BuboGPT is able to point out the specific location of an object in the image, when it is generating response or description for that object. Our contributions are two-fold: 1) An off-the-shelf visual grounding module based on SAM that extracts entities in a sentence and find corresponding masks in the image. 2) A two-stage training scheme and instruction dataset to endow joint text-image-audio understanding. Our experiments show that BuboGPT achieves impressive multi-modality understanding and visual grounding abilities during the interaction with human. It performs consistently well when provided by arbitrary modality combinations (either aligned or unaligned). Our code, model and dataset are available at https://bubo-gpt.github.io .

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 32 references that are in the map. First posted on arXiv 17 Jul 2023.

Builds on

28 of 32 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 27

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2023Segment Anythingcited by 8,158
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
15 more
  1. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  2. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  3. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  4. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  5. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  6. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  7. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  8. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  9. 2019Clotho: An Audio Captioning Datasetcited by 354
  10. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  11. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
  12. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  13. 2021Localizing Visual Sounds the Hard Waycited by 169
  14. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127
  15. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 53; 2025: 31; 2026: 6.

114 papers.

Sort:

By area:

  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  2. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  3. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  4. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  5. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  6. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  7. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  8. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  9. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  10. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  11. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  12. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
38 more
  1. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  2. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 125
  3. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 118
  4. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  5. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  6. 2024GroundingGPT:Language Enhanced Multi-modal Grounding Modelcited by 101
  7. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 101
  8. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  9. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  10. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  11. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  12. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  13. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  14. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  15. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 66
  16. 2024Multi-Object Hallucination in Vision-Language Modelscited by 60
  17. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  18. 2025GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensingcited by 55
  19. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Modelcited by 52
  20. 2024Toward Visual Grounding: A Surveycited by 52
  21. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policycited by 51
  22. 2024VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videoscited by 51
  23. 2023PG-Video-LLaVA: Pixel Grounding Large Video-Language Modelscited by 50
  24. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 50
  25. 2024Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMscited by 48
  26. 2025UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobilitycited by 48
  27. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  28. 2024DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLMcited by 45
  29. 2023See, Say, and Segment: Teaching LMMs to Overcome False Premisescited by 40
  30. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perceptioncited by 39
  31. 2024Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgerycited by 39
  32. 2025VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generationcited by 39
  33. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 38
  34. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  35. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  36. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  37. 2024F-LMM: Grounding Frozen Large Multimodal Modelscited by 33
  38. 2023Griffon: Spelling out All Object Locations at Any Granularity with Large Language Modelscited by 31

50 most cited of 114 All 114 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
  2. 2024Audio-visual training for improved grounding in video-text LLMs
  3. 2023Audio-Visual LLM for Video Understanding
  4. 2025Rethinking Visual Information Processing in Multimodal LLMs
  5. 2025MoDA: Modulation Adapter for Fine-Grained Visual Grounding in Instructional MLLMs
  6. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)
  7. 2024CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Models
  8. 2024LLMBind: A Unified Modality-Task Integration Framework