mlsanityData to 7 Oct 2026

MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning

arXiv 2409.20566 ↗First posted 30 Sep 2024cs.CV cs.CL cs.LGcited by 75

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

We present MM1.5, a new family of multimodal large language models (MLLMs) designed to enhance capabilities in text-rich image understanding, visual referring and grounding, and multi-image reasoning. Building upon the MM1 architecture, MM1.5 adopts a data-centric approach to model training, systematically exploring the impact of diverse data mixtures across the entire model training lifecycle. This includes high-quality OCR data and synthetic captions for continual pre-training, as well as an optimized visual instruction-tuning data mixture for supervised fine-tuning. Our models range from 1B to 30B parameters, encompassing both dense and mixture-of-experts (MoE) variants, and demonstrate that careful data curation and training strategies can yield strong performance even at small scales (1B and 3B). Additionally, we introduce two specialized variants: MM1.5-Video, designed for video understanding, and MM1.5-UI, tailored for mobile UI understanding. Through extensive empirical studies and ablations, we provide detailed insights into the training processes and decisions that inform our final designs, offering valuable guidance for future research in MLLM development.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 113 of its 194 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

113 of 194 references.

Closest ancestors · 32

Same direction, newest first

  1. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders28 Aug
  2. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Models16 Aug
  3. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  4. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phone3 Aug
  5. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models10 Jul
  6. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  7. 2024MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens17 Jun
  8. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding13 Jun
  9. 2024Dense Connector for MLLMs22 May
  10. 2024Efficient Multimodal Large Language Models: A Survey17 May
  11. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts9 May
  12. 2024MANTIS: Interleaved Multi-Image Instruction Tuning2 May
20 more
  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  2. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  3. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD9 Apr
  4. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward1 Apr
  5. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  6. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Models22 Feb
  7. 2024Efficient Multimodal Learning from Data-centric Perspective18 Feb
  8. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models8 Feb
  9. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  10. 2023Generative Multimodal Models are In-Context Learners20 Dec
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  12. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  13. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  14. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V17 Oct
  15. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  16. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  17. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  18. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs17 Jul
  19. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  20. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 81

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  12. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
38 more
  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  2. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  4. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  5. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  6. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  7. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  8. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  9. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  10. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  11. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  12. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  13. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  14. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  15. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  16. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  17. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  18. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  19. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  20. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  21. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  22. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  23. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  24. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  25. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 322
  26. 2024What matters when building vision-language models?cited by 321
  27. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  28. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  29. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  30. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
  31. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  32. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  33. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  34. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  35. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 193
  36. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  37. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 170
  38. 2024Building and better understanding vision-language models: insights and future directionscited by 169

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 47; 2026: 10.

75 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  4. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  5. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 296
  6. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  7. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 204
  8. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
  9. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 112
  10. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  11. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  12. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 56
38 more
  1. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  2. 2025Scaling Laws for Optimal Data Mixturescited by 49
  3. 2025Scaling Laws for Native Multimodal Modelscited by 47
  4. 2025Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decodingcited by 42
  5. 2025DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Searchcited by 41
  6. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  7. 2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generationcited by 32
  8. 2024ChatRex: Taming Multimodal LLM for Joint Perception and Understandingcited by 30
  9. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  10. 2025AToken: A Unified Tokenizer for Visioncited by 30
  11. 2025Scaling Vision Pre-Training to 4K Resolutioncited by 26
  12. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  13. 2025Ocean-OCR: Towards General OCR Application via a Vision-Language Modelcited by 24
  14. 2025StreamKV: Streaming Video Question-Answering with Segment-based KV Cache Retrieval and Compressioncited by 24
  15. 2025TikZero: Zero-Shot Text-Guided Graphics Program Synthesiscited by 21
  16. 2025Referring to Any Personcited by 20
  17. 2025Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understandingcited by 20
  18. 2025OmniCaptioner: One Captioner to Rule Them Allcited by 19
  19. 2025MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuningcited by 19
  20. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  21. 2025Time Blindness: Why Video-Language Models Can't See What Humans Can?cited by 17
  22. 2025MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizercited by 15
  23. 2024Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Modelscited by 14
  24. 2025MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGIcited by 14
  25. 2025Ferret-UI Lite: Lessons from Building Small On-Device GUI Agentscited by 14
  26. 2025A Token-level Text Image Foundation Model for Document Understandingcited by 13
  27. 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Executioncited by 13
  28. 2024POINTS1.5: Building a Vision-Language Model towards Real World Applicationscited by 12
  29. 2025Corvid: Improving Multimodal Large Language Models Towards Chain-of-Thought Reasoningcited by 11
  30. 2024From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impactscited by 10
  31. 2024DOGR: Towards Versatile Visual Document Grounding and Referringcited by 10
  32. 2024FlashSloth: Lightning Multimodal Large Language Models via Embedded Visual Compressioncited by 10
  33. 2025Aligned Better, Listen Better for Audio-Visual Large Language Modelscited by 10
  34. 2025Open-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal LLMs on Academic Resourcescited by 9
  35. 2025TACO: Think-Answer Consistency for Optimized Long-Chain Reasoning and Efficient Data Learning via Reinforcement Learning in LVLMscited by 9
  36. 2024INST-IT: Boosting Instance Understanding via Explicit Visual Prompt Instruction Tuningcited by 8
  37. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Reviewcited by 8
  38. 2025UniGen-1.5: Enhancing Image Generation and Editing through Reward Unification in Reinforcement Learningcited by 8

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024HERM: Benchmarking and Enhancing Multimodal LLMs for Human-Centric Understanding
  2. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding
  3. 2024The Revolution of Multimodal Large Language Models: A Survey
  4. 2023On the Performance of Multimodal Language Models
  5. 2024MM-Instruct: Generated Visual Instructions for Large Multimodal Model Alignment
  6. 2024MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct
  7. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Images
  8. 2024MM-InstructEval: Zero-Shot Evaluation of (Multimodal) Large Language Models on Multimodal Reasoning Tasks