mlsanityData to 7 Oct 2026

GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest

arXiv 2307.03601 ↗First posted 7 Jul 2023cs.CVcited by 300

Not on the map. Why?

Cited by 184 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Visual instruction tuning large language model(LLM) on image-text pairs has achieved general-purpose vision-language abilities. However, the lack of region-text pairs limits their advancements to fine-grained multimodal understanding. In this paper, we propose spatial instruction tuning, which introduces the reference to the region-of-interest(RoI) in the instruction. Before sending to LLM, the reference is replaced by RoI features and interleaved with language embeddings as a sequence. Our model GPT4RoI, trained on 7 region-text pair datasets, brings an unprecedented interactive and conversational experience compared to previous image-level models. (1) Interaction beyond language: Users can interact with our model by both language and drawing bounding boxes to flexibly adjust the referring granularity. (2) Versatile multimodal abilities: A variety of attribute information within each RoI can be mined by GPT4RoI, e.g., color, shape, material, action, etc. Furthermore, it can reason about multiple RoIs based on common sense. On the Visual Commonsense Reasoning(VCR) dataset, GPT4RoI achieves a remarkable accuracy of 81.6%, surpassing all existing models by a significant margin (the second place is 75.6%) and almost reaching human-level performance of 85.0%. The code and model can be found at https://github.com/jshilong/GPT4RoI.

Key facts. Cited by 300 papers in the map (data through 7 Oct 2026). Builds on 79 of its 125 references that are in the map. First posted on arXiv 7 Jul 2023.

Builds on

79 of 125 references.

References · 79

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  10. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2020End-to-End Object Detection with Transformerscited by 7,384
38 more
  1. 2017Mask R-CNNcited by 7,259
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  13. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  14. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  15. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  16. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  17. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  18. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  19. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  20. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  21. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  22. 2021Grounded Language-Image Pre-trainingcited by 1,147
  23. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  24. 2016Modeling Context in Referring Expressionscited by 1,054
  25. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  26. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  27. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  28. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  29. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  30. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  31. 2018Path Aggregation Network for Instance Segmentationcited by 796
  32. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  33. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  34. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  35. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  36. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  37. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  38. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 56; 2024: 128; 2025: 88; 2026: 28.

300 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  4. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  5. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  6. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  7. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  8. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  9. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  10. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  11. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  12. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
38 more
  1. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  2. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  3. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  4. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  5. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  6. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  7. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  8. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  9. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 220
  10. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  11. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208
  12. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 206
  13. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  14. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  15. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
  16. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 158
  17. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  18. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 153
  19. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  20. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  21. 2023TextDiffuser-2: Unleashing the Power of Language Models for Text Renderingcited by 136
  22. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  23. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  24. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  25. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  26. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  27. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
  28. 2024VISA: Reasoning Video Object Segmentation via Large Language Modelscited by 123
  29. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 118
  30. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  31. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  32. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  33. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videoscited by 99
  34. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 98
  35. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  36. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Modelscited by 96
  37. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 96
  38. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90

50 most cited of 300 All 300 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning
  2. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
  3. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data
  4. 2023SVIT: Scaling up Visual Instruction Tuning
  5. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction
  6. 2026Visual Instruction Tuning Aligns Modalities through Abstraction
  7. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning
  8. 2026Boosting Visual Instruction Tuning with Self-Supervised Guidance