mlsanityData to 7 Oct 2026

ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning

arXiv 2307.09474 ↗First posted 18 Jul 2023cs.CL cs.CVcited by 65

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Human-AI interactivity is a critical aspect that reflects the usability of multimodal large language models (MLLMs). However, existing end-to-end MLLMs only allow users to interact with them through language instructions, leading to the limitation of the interactive accuracy and efficiency. In this study, we present precise referring instructions that utilize diverse reference representations such as points and boxes as referring prompts to refer to the special region. This enables MLLMs to focus on the region of interest and achieve finer-grained interaction. Based on precise referring instruction, we propose ChatSpot, a unified end-to-end multimodal large language model that supports diverse forms of interactivity including mouse clicks, drag-and-drop, and drawing boxes, which provides a more flexible and seamless interactive experience. We also construct a multi-grained vision-language instruction-following dataset based on existing datasets and GPT-4 generating. Furthermore, we design a series of evaluation tasks to assess the effectiveness of region recognition and interaction. Experimental results showcase ChatSpot's promising performance.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 54 references that are in the map. First posted on arXiv 18 Jul 2023.

Builds on

38 of 54 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2017Deep reinforcement learning from human preferencescited by 3,846
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
24 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  4. 2022Emergent Abilities of Large Language Modelscited by 2,483
  5. 2019Towards VQA Models That Can Readcited by 1,696
  6. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  7. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  8. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  9. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  10. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  11. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  12. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  13. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  14. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  15. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  16. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  17. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  18. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  19. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
  20. 2019ICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text (RRC-ArT)cited by 147
  21. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  22. 2022Towards End-to-End Unified Scene Text Detection and Layout Analysiscited by 96
  23. 2021LaTr: Layout-Aware Transformer for Scene-Text VQAcited by 72
  24. 2020Point and Ask: Incorporating Pointing into Visual Question Answeringcited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 26; 2025: 20; 2026: 5.

65 papers.

Sort:

By area:

  1. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  2. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  3. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  4. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  5. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  6. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
  7. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  8. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  9. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  10. 2025Perception-R1: Pioneering Perception Policy with Reinforcement Learningcited by 116
  11. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  12. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 98
38 more
  1. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 95
  2. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  3. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  4. 2024VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLMcited by 71
  5. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 66
  6. 2024Focus Anywhere for Fine-grained Multi-page Document Understandingcited by 62
  7. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Modelscited by 59
  8. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 49
  9. 2023Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMscited by 47
  10. 2023Merlin:Empowering Multimodal LLMs with Foresight Mindscited by 46
  11. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  12. 2024Small Language Model Meets with Reinforced Vision Vocabularycited by 43
  13. 2024OneChart: Purify the Chart Structural Extraction via One Auxiliary Tokencited by 39
  14. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  15. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  16. 2025Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoningcited by 34
  17. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Modelscited by 33
  18. 2025Unhackable Temporal Rewarding for Scalable Video MLLMscited by 33
  19. 2023M3DBench: Let's Instruct Large Models with Multi-modal 3D Promptscited by 32
  20. 2023HallE-Control: Controlling Object Hallucination in Large Multimodal Modelscited by 31
  21. 2024Self-Supervised Visual Preference Alignmentcited by 31
  22. 2025Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Modelscited by 30
  23. 2024EarthMarker: A Visual Prompting Multi-modal Large Language Model for Remote Sensingcited by 29
  24. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 28
  25. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  26. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  27. 2024Evaluating Human-AI Collaboration: A Review and Methodological Frameworkcited by 25
  28. 2024Slow Perception: Let's Perceive Geometric Figures Step-by-stepcited by 24
  29. 2023Chain of Visual Perception: Harnessing Multimodal Large Language Models for Zero-shot Camouflaged Object Detectioncited by 22
  30. 2024The Evolution of Multimodal Model Architecturescited by 21
  31. 2025PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularitycited by 21
  32. 2023Vision-Language Instruction Tuning: A Review and Analysiscited by 17
  33. 2025Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Markscited by 17
  34. 2025Towards Better Dental AI: A Multimodal Benchmark and Instruction Dataset for Panoramic X-ray Analysiscited by 16
  35. 2024Rethinking Visual Prompting for Multimodal Large Language Models with External Knowledgecited by 13
  36. 2025Perception in Reflectioncited by 13
  37. 2024Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Modelscited by 12
  38. 2025VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMscited by 12

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024POEM: Interactive Prompt Optimization for Enhancing Multimodal Reasoning of Large Language Models
  2. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception
  3. 2023On the Performance of Multimodal Language Models
  4. 2024Enhancing Model Performance: Another Approach to Vision-Language Instruction Tuning
  5. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
  6. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  7. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs
  8. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst