mlsanityData to 7 Oct 2026

InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

arXiv 2305.06500 ↗First posted 11 May 2023cs.CV cs.LGcited by 2,855

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large-scale pre-training and instruction tuning have been successful at creating general-purpose language models with broad competence. However, building general-purpose vision-language models is challenging due to the rich input distributions and task diversity resulting from the additional visual input. Although vision-language pretraining has been widely studied, vision-language instruction tuning remains under-explored. In this paper, we conduct a systematic and comprehensive study on vision-language instruction tuning based on the pretrained BLIP-2 models. We gather 26 publicly available datasets, covering a wide variety of tasks and capabilities, and transform them into instruction tuning format. Additionally, we introduce an instruction-aware Query Transformer, which extracts informative features tailored to the given instruction. Trained on 13 held-in datasets, InstructBLIP attains state-of-the-art zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and larger Flamingo models. Our models also lead to state-of-the-art performance when finetuned on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA questions with image contexts). Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over concurrent multimodal models. All InstructBLIP models are open-sourced at https://github.com/salesforce/LAVIS/tree/main/projects/instructblip.

Key facts. Cited by 2,855 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 52 references that are in the map. First posted on arXiv 11 May 2023.

Builds on

43 of 52 references.

References · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
31 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  7. 2019Towards VQA Models That Can Readcited by 1,696
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  10. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  12. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  13. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  14. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  15. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  16. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  17. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  18. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  19. 2018nocaps: novel object captioning at scalecited by 444
  20. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  21. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  22. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  23. 2016Visual Dialogcited by 377
  24. 2022Visual Spatial Reasoningcited by 288
  25. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  26. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  27. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  28. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  29. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  30. 2019Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baselinecited by 60
  31. 2022LAVIS: A Library for Language-Vision Intelligencecited by 55

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 338; 2024: 1081; 2025: 839; 2026: 597.

2,855 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  12. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
38 more
  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  3. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  4. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  6. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  7. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  8. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  9. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  10. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  12. 2023A Survey on Multimodal Large Language Modelscited by 898
  13. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  14. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  15. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  16. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  17. 2025Qwen2.5-Omni Technical Reportcited by 704
  18. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  19. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  20. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  21. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  22. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  23. 2023A Comprehensive Overview of Large Language Modelscited by 639
  24. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  25. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  26. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  27. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  28. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  29. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  30. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  31. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  32. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  33. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  34. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  35. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  36. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  37. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  38. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492

50 most cited of 2,855 All 2,855 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Rethinking Overlooked Aspects in Vision-Language Models
  2. 2024Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning
  3. 2026Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Following
  4. 2025Do we Really Need Visual Instructions? Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models
  5. 2023Vision-Language Instruction Tuning: A Review and Analysis
  6. 2026MM-IFEval-Pro: A Multilingual and Attack-Resistant Benchmark for Instruction-Following in Vision-Language Models
  7. 2026Boosting Visual Instruction Tuning with Self-Supervised Guidance
  8. 2024POINTS: Improving Your Vision-language Model with Affordable Strategies