mlsanityData to 7 Oct 2026

Visual Instruction Tuning

arXiv 2304.08485 ↗First posted 17 Apr 2023cs.CV cs.AI cs.CLcited by 7,792

Not on the map. Why?

Cited by 3,450 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Instruction tuning large language models (LLMs) using machine-generated instruction-following data has improved zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field. In this paper, we present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding.Our early experiments show that LLaVA demonstrates impressive multimodel chat abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen images/instructions, and yields a 85.1% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset. When fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of 92.53%. We make GPT-4 generated visual instruction tuning data, our model and code base publicly available.

Key facts. Cited by 7,792 papers in the map (data through 7 Oct 2026). Builds on 52 of its 63 references that are in the map. First posted on arXiv 17 Apr 2023.

Builds on

52 of 63 references.

References · 52

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2023Segment Anythingcited by 8,158
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  8. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2022Visual Prompt Tuningcited by 1,674
  12. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  13. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  14. 2021Grounded Language-Image Pre-trainingcited by 1,147
  15. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  16. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  17. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  18. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  19. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  20. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  21. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  22. 2022Language-driven Semantic Segmentationcited by 711
  23. 2023Instruction Tuning with GPT-4cited by 676
  24. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  25. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  26. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  27. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  28. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  29. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  30. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  31. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  32. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  33. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  34. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  35. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  36. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 197
  37. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  38. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 585; 2024: 2213; 2025: 2931; 2026: 2063.

7,792 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  9. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
38 more
  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  3. 2025Gemma 3 Technical Reportcited by 1,686
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  5. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  7. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  8. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  9. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  10. 2024Depth Anything V2cited by 1,385
  11. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  12. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  13. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  14. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  15. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  16. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  17. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  18. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  19. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  20. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  21. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  22. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  23. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  24. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  25. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  26. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  27. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  28. 2023A Survey on Multimodal Large Language Modelscited by 898
  29. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  30. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  31. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  32. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  33. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  34. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  35. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  36. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  37. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  38. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749

50 most cited of 7,792 All 7,792 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
  2. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction
  3. 2024Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data
  4. 2025Do we Really Need Visual Instructions? Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models
  5. 2024Rethinking Overlooked Aspects in Vision-Language Models
  6. 2024M2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning
  7. 2024LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering
  8. 2024Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning