mlsanityData to 7 Oct 2026

MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale

arXiv 2412.05237 ↗First posted 6 Dec 2024cs.CL cs.CVcited by 125

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Open-source multimodal large language models (MLLMs) have shown significant potential in a broad range of multimodal tasks. However, their reasoning capabilities remain constrained by existing instruction-tuning datasets, which were predominately repurposed from academic datasets such as VQA, AI2D, and ChartQA. These datasets target simplistic tasks, and only provide phrase-level answers without any intermediate rationales. To address these challenges, we introduce a scalable and cost-effective method to construct a large-scale multimodal instruction-tuning dataset with rich intermediate rationales designed to elicit CoT reasoning. Using only open models, we create a dataset containing 12M instruction-response pairs to cover diverse, reasoning-intensive tasks with detailed and faithful rationales. Experiments demonstrate that training MLLMs on this dataset significantly improves reasoning capabilities, achieving state-of-the-art performance on benchmarks such as MathVerse (+8.1%), MMMU-Pro (+7%), and MuirBench (+13.3%). Additionally, the model demonstrates notable improvements of up to 4% on non-reasoning-based benchmarks. Ablation studies further highlight the importance of key components, such as rewriting and self-filtering, in the dataset construction process.

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 104 of its 145 references that are in the map. First posted on arXiv 6 Dec 2024.

Builds on

104 of 145 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Step15 Nov
  2. 2024Improve Vision Language Model Chain-of-thought Reasoning21 Oct
  3. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  4. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs27 Nov
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  7. 2023DetGPT: Detect What You Need via Reasoning23 May
  8. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Models23 May
  9. 2022Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning1 Dec
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep
  11. 2022CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoning10 Aug
  12. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning19 Mar
1 more
  1. 2022The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning10 Feb

Other references · 91

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
38 more
  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  3. 2015VQA: Visual Question Answeringcited by 2,489
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  5. 2019Towards VQA Models That Can Readcited by 1,696
  6. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  7. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  9. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  10. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  11. 2016Modeling Context in Referring Expressionscited by 1,054
  12. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  13. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  14. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  15. 2016A Diagram Is Worth A Dozen Imagescited by 793
  16. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  17. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  18. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  19. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  20. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  21. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  22. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  23. 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 489
  24. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  25. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  26. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  27. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  28. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  29. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  30. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  31. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  32. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  33. 2019Scene Text Visual Question Answeringcited by 327
  34. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  35. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  36. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  37. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  38. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 85; 2026: 39.

125 papers.

Sort:

By area:

  1. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  2. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
  3. 2025Kimi-VL Technical Reportcited by 381
  4. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 364
  5. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  6. 2025MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learningcited by 294
  7. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 267
  8. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  9. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  10. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  11. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
  12. 2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cyclescited by 122
38 more
  1. 2025Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decodingcited by 101
  2. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 94
  3. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
  4. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
  5. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 78
  6. 2024FastVLM: Efficient Vision Encoding for Vision Language Modelscited by 76
  7. 2025A Survey on Diffusion Language Modelscited by 75
  8. 2025Unlocking Multimodal Mathematical Reasoning via Process Reward Modelcited by 64
  9. 2025Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Modelcited by 60
  10. 2025Look-Back: Implicit Visual Re-focusing in MLLM Reasoningcited by 54
  11. 2025Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learningcited by 51
  12. 2025FineVision: Open Data Is All You Needcited by 51
  13. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generationcited by 50
  14. 2025MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoningcited by 45
  15. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  16. 2025BitVLA: 1-bit Vision-Language-Action Models for Robotics Manipulationcited by 42
  17. 2025Advancing Multimodal Reasoning via Reinforcement Learning with Cold Startcited by 41
  18. 2025MathCanvas: Intrinsic Visual Chain-of-Thought for Multimodal Mathematical Reasoningcited by 35
  19. 2025VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Searchcited by 34
  20. 2025First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Trainingcited by 34
  21. 2025Point-RFT: Improving Multimodal Reasoning with Visually Grounded Reinforcement Finetuningcited by 31
  22. 2025X-Reasoner: Towards Generalizable Reasoning Across Modalities and Domainscited by 30
  23. 2025Mitigating Visual Forgetting via Take-along Visual Conditioning for Multi-modal Long CoT Reasoningcited by 28
  24. 2026ABot-N0: Technical Report on the VLA Foundation Model for Versatile Embodied Navigationcited by 23
  25. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 22
  26. 2025Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Modelscited by 21
  27. 2025MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddingscited by 20
  28. 2025RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learningcited by 20
  29. 2025video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Modelcited by 19
  30. 2025Multi-Step Visual Reasoning with Visual Tokens Scaling and Verificationcited by 19
  31. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  32. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 19
  33. 2025Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbonecited by 18
  34. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real Worldcited by 16
  35. 2025Growing a Multi-head Twig via Distillation and Reinforcement Learning to Accelerate Large Vision-Language Modelscited by 15
  36. 2025Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Schemecited by 14
  37. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  38. 2025Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Large Vision-Language Modelscited by 14

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Training-Free Mitigation of Language Reasoning Degradation After Multimodal Instruction Tuning
  2. 2026OmniThoughtVis: A Scalable Distillation Pipeline for Deployable Multimodal Reasoning Models
  3. 2025MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models
  4. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
  5. 2024CoMMIT: Coordinated Multimodal Instruction Tuning
  6. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization
  7. 2023Genixer: Empowering Multimodal Large Language Models as a Powerful Data Generator
  8. 2025MathSE: Improving Multimodal Mathematical Reasoning via Self-Evolving Iterative Reflection and Reward-Guided Fine-Tuning