mlsanityData to 14 Oct 2026

MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training

arXiv 2403.09611 ↗First posted 14 Mar 2024cs.CV cs.CL cs.LGcited by 232

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

In this work, we discuss building performant Multimodal Large Language Models (MLLMs). In particular, we study the importance of various architecture components and data choices. Through careful and comprehensive ablations of the image encoder, the vision language connector, and various pre-training data choices, we identified several crucial design lessons. For example, we demonstrate that for large-scale multimodal pre-training using a careful mix of image-caption, interleaved image-text, and text-only data is crucial for achieving state-of-the-art (SOTA) few-shot results across multiple benchmarks, compared to other published pre-training results. Further, we show that the image encoder together with image resolution and the image token count has substantial impact, while the vision-language connector design is of comparatively negligible importance. By scaling up the presented recipe, we build MM1, a family of multimodal models up to 30B parameters, including both dense models and mixture-of-experts (MoE) variants, that are SOTA in pre-training metrics and achieve competitive performance after supervised fine-tuning on a range of established multimodal benchmarks. Thanks to large-scale pre-training, MM1 enjoys appealing properties such as enhanced in-context learning, and multi-image reasoning, enabling few-shot chain-of-thought prompting.

Key facts. Cited by 232 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 115 of its 132 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

115 of 132 references.

Closest ancestors · 22

Same direction, newest first

  1. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Models22 Feb
  2. 2024Efficient Multimodal Learning from Data-centric Perspective18 Feb
  3. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models8 Feb
  4. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  5. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  6. 2023Generative Multimodal Models are In-Context Learners20 Dec
  7. 2023VILA: On Pre-training for Visual Language Models12 Dec
  8. 2023Honeybee: Locality-enhanced Projector for Multimodal LLM11 Dec
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  10. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  11. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  12. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents9 Nov
10 more
  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  4. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  8. 2023Generating Images with Multimodal Language Models26 May
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  10. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 93

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2023GPT-4 Technical Reportcited by 18,396
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  10. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  13. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  14. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  15. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  16. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  17. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  18. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  19. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  20. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  21. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  22. 2019Towards VQA Models That Can Readcited by 1,718
  23. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  24. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  25. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  26. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  27. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  28. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  29. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  30. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  31. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  32. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  33. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  34. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  35. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  36. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  37. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  38. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867

50 most cited of 93 All 93 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 158; 2025: 60; 2026: 14.

232 papers.

Sort:

By area:

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  3. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  5. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  6. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  7. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  8. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  9. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  10. 2024What matters when building vision-language models?cited by 326
  11. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  12. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 292
38 more
  1. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
  2. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  3. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 216
  4. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 207
  5. 2024Programming Refusal with Conditional Activation Steeringcited by 206
  6. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
  7. 2024An Introduction to Vision-Language Modelingcited by 178
  8. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 176
  9. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 176
  10. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  11. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 172
  12. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 159
  13. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 153
  14. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 152
  15. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 148
  16. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 147
  17. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 127
  18. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 125
  19. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  20. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Modelscited by 113
  21. 2024Multimodal Autoregressive Pre-training of Large Vision Encoderscited by 109
  22. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  23. 2024Why are Visually-Grounded Language Models Bad at Image Classification?cited by 102
  24. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  25. 2024Efficient Multimodal Large Language Models: A Surveycited by 96
  26. 2024Unveiling Encoder-Free Vision-Language Modelscited by 94
  27. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 88
  28. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  29. 2024Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Modelscited by 76
  30. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 76
  31. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  32. 2024DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Modelscited by 75
  33. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 73
  34. 2024ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Areacited by 71
  35. 2024Enhancing Large Vision Language Models with Self-Training on Image Comprehensioncited by 70
  36. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Textcited by 59
  37. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 59
  38. 2024Needle In A Multimodal Haystackcited by 58

50 most cited of 232 All 232 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
  2. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Images
  3. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
  4. 2026Multimodal Large Language Models as Image Classifiers
  5. 2024MM-LLMs: Recent Advances in MultiModal Large Language Models
  6. 2024Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning
  7. 2023On the Performance of Multimodal Language Models
  8. 2026How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining