mlsanityData to 7 Oct 2026

SVIT: Scaling up Visual Instruction Tuning

arXiv 2307.04087 ↗First posted 9 Jul 2023cs.CVcited by 161

Not on the map. Why?

Cited by 124 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Thanks to the emerging of foundation models, the large language and vision models are integrated to acquire the multimodal ability of visual captioning, question answering, etc. Although existing multimodal models present impressive performance of visual understanding and reasoning, their limits are still largely under-explored due to the scarcity of high-quality instruction tuning data. To push the limits of multimodal capability, we Scale up Visual Instruction Tuning (SVIT) by constructing a dataset of 4.2 million visual instruction tuning data including 1.6M conversation question-answer (QA) pairs, 1.6M complex reasoning QA pairs, 1.0M referring QA pairs and 106K detailed image descriptions. Besides the volume, the proposed dataset is also featured by the high quality and rich diversity, which is generated by prompting GPT-4 with the abundant manual annotations of images. We also propose a new data recipe to select subset with better diversity and balance, which evokes model's superior capabilities. Extensive experiments verify that SVIT-v1.5, trained on the proposed dataset, outperforms state-of-the-art Multimodal Large Language Models on popular benchmarks. The data and code are publicly available at https://github.com/BAAI-DCAI/Visual-Instruction-Tuning.

Key facts. Cited by 161 papers in the map (data through 7 Oct 2026). Builds on 39 of its 55 references that are in the map. First posted on arXiv 9 Jul 2023.

Builds on

39 of 55 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
27 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2019Towards VQA Models That Can Readcited by 1,696
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  11. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  12. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  13. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  14. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  15. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  16. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  17. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  18. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  19. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  20. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  21. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  22. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  23. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  24. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  25. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  26. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  27. 2023InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Languagecited by 96

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 83; 2025: 38; 2026: 16.

161 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  5. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  6. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  7. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  8. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  9. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  10. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  11. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  12. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
38 more
  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  2. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  3. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  4. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  5. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  6. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  7. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  8. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  9. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
  10. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  11. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  12. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  13. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 128
  14. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
  15. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
  16. 2024Efficient Multimodal Learning from Data-centric Perspectivecited by 120
  17. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  18. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  19. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  20. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  21. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  22. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  23. 2024Unveiling Encoder-Free Vision-Language Modelscited by 90
  24. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 89
  25. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84
  26. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  27. 2024Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Modelscited by 75
  28. 2024Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMscited by 73
  29. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  30. 2024Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Roboticscited by 68
  31. 2023MM-LIMA: Less Is More for Alignment in Multi-Modal Datasetscited by 61
  32. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Textcited by 58
  33. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 56
  34. 2024RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesiscited by 55
  35. 2024Multi-modal Situated Reasoning in 3D Scenescited by 50
  36. 2023Mitigating Hallucination in Visual Language Models with Visual Supervisioncited by 49
  37. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMscited by 46
  38. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 43

50 most cited of 161 All 161 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
  2. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models
  3. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data
  4. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest
  5. 2026Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Following
  6. 2023On the Performance of Multimodal Language Models
  7. 2025Learning to Instruct for Visual Instruction Tuning
  8. 2025Visual Compositional Tuning