mlsanityData to 7 Oct 2026

VILA: On Pre-training for Visual Language Models

arXiv 2312.07533 ↗First posted 12 Dec 2023cs.CVcited by 686

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Visual language models (VLMs) rapidly progressed with the recent success of large language models. There have been growing efforts on visual instruction tuning to extend the LLM with visual inputs, but lacks an in-depth study of the visual language pre-training process, where the model learns to perform joint modeling on both modalities. In this work, we examine the design options for VLM pre-training by augmenting LLM towards VLM through step-by-step controllable comparisons. We introduce three main findings: (1) freezing LLMs during pre-training can achieve decent zero-shot performance, but lack in-context learning capability, which requires unfreezing the LLM; (2) interleaved pre-training data is beneficial whereas image-text pairs alone are not optimal; (3) re-blending text-only instruction data to image-text data during instruction fine-tuning not only remedies the degradation of text-only tasks, but also boosts VLM task accuracy. With an enhanced pre-training recipe we build VILA, a Visual Language model family that consistently outperforms the state-of-the-art models, e.g., LLaVA-1.5, across main benchmarks without bells and whistles. Multi-modal pre-training also helps unveil appealing properties of VILA, including multi-image reasoning, enhanced in-context learning, and better world knowledge.

Key facts. Cited by 686 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 57 of its 74 references that are in the map. First posted on arXiv 12 Dec 2023.

Builds on

57 of 74 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  2. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  3. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  4. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  5. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning14 Sep
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  7. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  9. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  11. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  12. 2020Measuring Massive Multitask Language Understandingcited by 7,768
34 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  13. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  14. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  15. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  16. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  17. 2019Towards VQA Models That Can Readcited by 1,696
  18. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  19. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  20. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  21. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  22. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  23. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  24. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  25. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  26. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  27. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  28. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  29. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  30. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  31. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  32. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  33. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Modelscited by 100
  34. 2023Jointly Training Large Autoregressive Multimodal Modelscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 222; 2025: 317; 2026: 146.

686 papers.

Sort:

By area:

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  3. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  4. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  5. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  6. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  7. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  8. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  9. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  10. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  11. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  12. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
38 more
  1. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  2. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
  3. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  4. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  5. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 331
  6. 2024What matters when building vision-language models?cited by 321
  7. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  8. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  9. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  10. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  11. 2025Improving Video Generation with Human Feedbackcited by 267
  12. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  13. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  14. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 235
  15. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  16. 2025ImgEdit: A Unified Image Editing Dataset and Benchmarkcited by 225
  17. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  18. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
  19. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  20. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 206
  21. 2023Video Understanding with Large Language Models: A Surveycited by 203
  22. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  23. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 178
  24. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  25. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  26. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156
  27. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstructioncited by 154
  28. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 153
  29. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 152
  30. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  31. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  32. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  33. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  34. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 134
  35. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  36. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  37. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  38. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 131

50 most cited of 686 All 686 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning
  2. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
  3. 2025Should VLMs be Pre-trained with Image Data?
  4. 2025Do we Really Need Visual Instructions? Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models
  5. 2023Vision-Language Instruction Tuning: A Review and Analysis
  6. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
  7. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
  8. 2025Visual Large Language Models for Generalized and Specialized Applications