mlsanityData to 7 Oct 2026

Vision-Language Instruction Tuning: A Review and Analysis

arXiv 2311.08172 ↗First posted 14 Nov 2023cs.MM cs.CVcited by 17

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Instruction tuning is a crucial supervised training phase in Large Language Models (LLMs), aiming to enhance the LLM's ability to generalize instruction execution and adapt to user preferences. With the increasing integration of multi-modal data into LLMs, there is growing interest in Vision-Language Instruction Tuning (VLIT), which presents more complex characteristics compared to pure text instruction tuning. In this paper, we systematically review the latest VLIT settings and corresponding datasets in multi-modal LLMs and provide insights into the intrinsic motivations behind their design. For the first time, we offer a detailed multi-perspective categorization for existing VLIT datasets and identify the characteristics that high-quality VLIT data should possess. By incorporating these characteristics as guiding principles into the existing VLIT data construction process, we conduct extensive experiments and verify their positive impact on the performance of tuned multi-modal LLMs. Furthermore, we discuss the current challenges and future research directions of VLIT, providing insights for the continuous development of this field. The code and dataset related to this paper have been open-sourced at https://github.com/palchenli/VL-Instruction-Tuning.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 101 of its 123 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

101 of 123 references.

Closest ancestors · 20

Same direction, newest first

  1. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  2. 2023What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning2 Nov
  3. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  4. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  5. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Models19 Sep
  6. 2023An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models18 Sep
  7. 2023Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethics13 Sep
  8. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  9. 2023MLLM-DataEngine: An Iterative Refinement Approach for MLLM25 Aug
  10. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models25 Aug
  11. 2023VIGC: Visual Instruction Generation and Correction24 Aug
  12. 2023MM-LIMA: Less Is More for Alignment in Multi-Modal Datasets23 Aug
8 more
  1. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data20 Aug
  2. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  3. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning18 Jul
  4. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs17 Jul
  5. 2023Visual Instruction Tuning with Polite Flamingo3 Jul
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  7. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration15 Jun
  8. 2023LMEye: An Interactive Perception Network for Large Language Models5 May

Other references · 81

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  10. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  11. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  12. 2023Visual Instruction Tuningcited by 7,792
38 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2015VQA: Visual Question Answeringcited by 2,489
  12. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  13. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  14. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  15. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  16. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  17. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  18. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  19. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  20. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  21. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  22. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  23. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  24. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  25. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  26. 2022MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessmentcited by 514
  27. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  28. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  29. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  30. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  31. 2022Multimodal Learning with Transformers: A Surveycited by 399
  32. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  33. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  34. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  35. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  36. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  37. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  38. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 5; 2026: 4.

17 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  3. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  4. 2024An Introduction to Vision-Language Modelingcited by 175
  5. 2025Exploring the Underwater World Segmentation without Extra Trainingcited by 16
  6. 2024MetaToken: Detecting Hallucination in Image Descriptions by Meta Classificationcited by 13
  7. 2024Frontiers in Intelligent Colonoscopycited by 13
  8. 2024RS-GPT4V: A Unified Multimodal Instruction-Following Dataset for Remote Sensing Image Understandingcited by 9
  9. 2025Re-Imagining Multimodal Instruction Tuning: A Representation Viewcited by 9
  10. 2025MLLM-Selector: Necessity and Diversity-driven High-Value Data Selection for Enhanced Visual Instruction Tuningcited by 9
  11. 2025ClinKD: Cross-Modal Clinical Knowledge Distiller For Multi-Task Medical Imagescited by 2
  12. 2025Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Modelscited by 1
5 more
  1. 2026Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLMcited by 1
  2. 2026Does the Question Really Matter? Training-Free Data Selection for Vision-Language SFTcited by 1
  3. 2026RISE: Reliable Improvement in Self-Evolving Vision-Language Modelscited by 1
  4. 2024C3L: Content Correlated Vision-Language Instruction Tuning Data Generation via Contrastive Learningcited by 0
  5. 2026BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor Attackscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Instruction Tuning with and without Context: Behavioral Shifts and Downstream Impact
  2. 2024Rethinking Overlooked Aspects in Vision-Language Models
  3. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
  4. 2024LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering
  5. 2025Instruction-Following Evaluation of Large Vision-Language Models
  6. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning
  7. 2025Cream of the Crop: Harvesting Rich, Scalable and Transferable Multi-Modal Data for Instruction Fine-Tuning
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning