mlsanityData to 14 Oct 2026

Improved Baselines with Visual Instruction Tuning

arXiv 2310.03744 ↗First posted 5 Oct 2023cs.CV cs.AI cs.CLcited by 4,418

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large multimodal models (LMM) have recently shown encouraging progress with visual instruction tuning. In this note, we show that the fully-connected vision-language cross-modal connector in LLaVA is surprisingly powerful and data-efficient. With simple modifications to LLaVA, namely, using CLIP-ViT-L-336px with an MLP projection and adding academic-task-oriented VQA data with simple response formatting prompts, we establish stronger baselines that achieve state-of-the-art across 11 benchmarks. Our final 13B checkpoint uses merely 1.2M publicly available data, and finishes full training in ~1 day on a single 8-A100 node. We hope this can make state-of-the-art LMM research more accessible. Code and model will be publicly available.

Key facts. Cited by 4,418 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 71 references that are in the map. First posted on arXiv 5 Oct 2023.

Builds on

50 of 71 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  2. 2023An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models18 Sep
  3. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use12 Aug
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  5. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  7. 2023Visual Instruction Tuning with Polite Flamingo3 Jul
  8. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
28 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  4. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  7. 2019Towards VQA Models That Can Readcited by 1,718
  8. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  10. 2023LIMA: Less Is More for Alignmentcited by 1,100
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  12. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  13. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  14. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  15. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  16. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  17. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  18. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  19. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  20. 2023Are aligned neural networks adversarially aligned?cited by 315
  21. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  22. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
  23. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  24. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  25. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  26. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  27. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  28. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 166; 2024: 1427; 2025: 1691; 2026: 1134.

4,418 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  7. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  8. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  9. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  10. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  11. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  12. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
38 more
  1. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  2. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  3. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  4. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  5. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  6. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  7. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  8. 2025Qwen2.5-Omni Technical Reportcited by 736
  9. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  10. 2024Yi: Open Foundation Models by 01.AIcited by 719
  11. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  12. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  13. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  14. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  15. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  16. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  17. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
  18. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  19. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  20. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  21. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  22. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  23. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  24. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  25. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 530
  26. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 526
  27. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  28. 2024Long Context Transfer from Language to Visioncited by 515
  29. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  30. 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 503
  31. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 490
  32. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  33. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  34. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  35. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  36. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  37. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  38. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415

50 most cited of 4,418 All 4,418 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
  2. 2024Enhancing Model Performance: Another Approach to Vision-Language Instruction Tuning
  3. 2024Instruction Tuning-free Visual Token Complement for Multimodal LLMs
  4. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models
  5. 2024LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning
  6. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Data
  7. 2025Efficiently Integrate Large Language Models with Visual Perception: A Survey from the Training Paradigm Perspective
  8. 2024CROME: Cross-Modal Adapters for Efficient Multimodal LLM