mlsanityData to 14 Oct 2026

To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning

arXiv 2311.07574 ↗First posted 13 Nov 2023cs.CVcited by 144

Not on the map. Why?

Cited by 106 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Existing visual instruction tuning methods typically prompt large language models with textual descriptions to generate instruction-following data. Despite the promising performance achieved, these descriptions are derived from image annotations, which are oftentimes coarse-grained. Furthermore, the instructions might even contradict the visual content without observing the entire visual context. To address this challenge, we introduce a fine-grained visual instruction dataset, LVIS-Instruct4V, which contains 220K visually aligned and context-aware instructions produced by prompting the powerful GPT-4V with images from LVIS. Through experimental validation and case studies, we demonstrate that high-quality visual instructional data could improve the performance of LLaVA-1.5, a state-of-the-art large multimodal model, across a wide spectrum of benchmarks by clear margins. Notably, by simply replacing the LLaVA-Instruct with our LVIS-Instruct4V, we achieve better results than LLaVA on most challenging LMM benchmarks, e.g., LLaVAw (76.7 vs. 70.7) and MM-Vet (40.2 vs. 35.4). We release our data and model at https://github.com/X2FD/LVIS-INSTRUCT4V.

Key facts. Cited by 144 papers in the map (data through 14 Oct 2026). Builds on 40 of its 49 references that are in the map. First posted on arXiv 13 Nov 2023.

Builds on

40 of 49 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
28 more
  1. 2023Qwen Technical Reportcited by 3,344
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2015VQA: Visual Question Answeringcited by 2,506
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  10. 2019Towards VQA Models That Can Readcited by 1,718
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  12. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  13. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  14. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  15. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  16. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  17. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  18. 2016Modeling Context in Referring Expressionscited by 1,063
  19. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  20. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  21. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  22. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  23. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  24. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  25. 2016Image Captioning with Semantic Attentioncited by 260
  26. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  27. 2016Boosting Image Captioning with Attributescited by 76
  28. 2015Where To Look: Focus Regions for Visual Question Answeringcited by 64

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 77; 2025: 47; 2026: 16.

144 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  3. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  4. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  5. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  6. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  7. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
  8. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  9. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  10. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  11. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  12. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
38 more
  1. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
  2. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 205
  3. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 197
  4. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  5. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 174
  6. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  7. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 159
  8. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  9. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 139
  10. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 131
  11. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 127
  12. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 122
  13. 2024Efficient Multimodal Learning from Data-centric Perspectivecited by 122
  14. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 119
  15. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  16. 2024Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inferencecited by 97
  17. 2024Efficient Multimodal Large Language Models: A Surveycited by 96
  18. 2024Unveiling Encoder-Free Vision-Language Modelscited by 94
  19. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  20. 2024DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMscited by 75
  21. 2024Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMscited by 74
  22. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perceptioncited by 73
  23. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 70
  24. 2024Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Datacited by 67
  25. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Textcited by 59
  26. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsificationcited by 59
  27. 2024From Pixels to Prose: A Large Dataset of Dense Image Captionscited by 54
  28. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 53
  29. 2025FineVision: Open Data Is All You Needcited by 53
  30. 2024Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usagecited by 51
  31. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  32. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 45
  33. 2024VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Modelscited by 43
  34. 2025EVEv2: Improved Baselines for Encoder-Free Vision-Language Modelscited by 41
  35. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracycited by 39
  36. 2024SOLO: A Single Transformer for Scalable Vision-Language Modelingcited by 37
  37. 2024Open-FinLLMs: Open Multimodal Large Language Models for Financial Applicationscited by 36
  38. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36

50 most cited of 144 All 144 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest
  2. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
  3. 2025Learning to Instruct for Visual Instruction Tuning
  4. 2024Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning
  5. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data
  6. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models
  7. 2025Do we Really Need Visual Instructions? Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models
  8. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning