mlsanityData to 7 Oct 2026

M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought

arXiv 2405.16473 ↗First posted 26 May 2024cs.CV cs.AI cs.CLcited by 120

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Multi-modal Chain-of-Thought (MCoT) requires models to leverage knowledge from both textual and visual modalities for step-by-step reasoning, which gains increasing attention. Nevertheless, the current MCoT benchmark still faces some challenges: (1) absence of visual modal reasoning, (2) single-step visual modal reasoning, and (3) Domain missing, thereby hindering the development of MCoT. Motivated by this, we introduce a novel benchmark (M3CoT) to address the above challenges, advancing the multi-domain, multi-step, and multi-modal CoT. Additionally, we conduct a thorough evaluation involving abundant MCoT approaches on Vision Large Language Models (VLLMs). In addition, we highlight that the current VLLMs still struggle to correctly reason in M3CoT and there remains a large gap between existing VLLMs and human performance in M3CoT, despite their superior results on previous MCoT benchmarks. To our knowledge, we take the first meaningful step toward the multi-domain, multi-step, and multi-modal scenario in MCoT. We hope that M3CoT can serve as a valuable resource, providing a pioneering foundation in multi-domain, multi-step, multi-modal chain-of-thought research.

Key facts. Cited by 120 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 50 references that are in the map. First posted on arXiv 26 May 2024.

Builds on

31 of 50 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023Multi-modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models14 Dec
  2. 2023Assessing GPT4-V on Structured Reasoning Tasks13 Dec
  3. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Models27 Nov
  4. 2023Boosting the Power of Small Multimodal Reasoning Models to Match Larger Models with Self-Consistency Training23 Nov
  5. 2023The Role of Chain-of-Thought in Complex Vision-Language Reasoning Task15 Nov
  6. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models25 Oct
  7. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  8. 2023Enhancing Human-like Multi-Modal Reasoning: A New Challenging Dataset and Comprehensive Framework24 Jul
  9. 2023T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answering5 May
  10. 2022Abstract Visual Reasoning with Tangram Shapes29 Nov
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep
  12. 2022The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning10 Feb

Other references · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  9. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  11. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  12. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
7 more
  1. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  2. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  3. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  4. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  5. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  6. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 175
  7. 2023Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languagescited by 96

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 67; 2026: 32.

120 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  3. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  4. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  5. 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understandingcited by 147
  6. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
  7. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 123
  8. 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agentcited by 114
  9. 2025Visual Planning: Let's Think Only with Imagescited by 85
  10. 2025Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Modelcited by 60
  11. 2024Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thoughtcited by 56
  12. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
38 more
  1. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 54
  2. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  3. 2024Interleaved-Modal Chain-of-Thoughtcited by 47
  4. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  5. 2025VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoningcited by 41
  6. 2025Advancing Multimodal Reasoning via Reinforcement Learning with Cold Startcited by 41
  7. 2025Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thoughtcited by 34
  8. 2024What Factors Affect Multi-Modal In-Context Learning? An In-Depth Explorationcited by 31
  9. 2024Diving into Self-Evolving Training for Multimodal Reasoningcited by 31
  10. 2024CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Modelscited by 29
  11. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  12. 2025More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Modelscited by 26
  13. 2025Mind with Eyes: from Language Reasoning to Multimodal Reasoningcited by 22
  14. 2025Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmarkcited by 20
  15. 2024AutoCAP: Towards Automatic Cross-lingual Alignment Planning for Zero-shot Chain-of-Thoughtcited by 19
  16. 2024HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Taskscited by 19
  17. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  18. 2024Enhancing Advanced Visual Reasoning Ability of Large Language Modelscited by 18
  19. 2025ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in Large Language Modelscited by 18
  20. 2025EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoTcited by 16
  21. 2024VCR: A Task for Pixel-Level Complex Reasoning in Vision Language Models via Restoring Occluded Textcited by 15
  22. 2025Co-Reinforcement Learning for Unified Multimodal Understanding and Generationcited by 15
  23. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  24. 2024What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practicescited by 14
  25. 2025GRE Suite: Geo-localization Inference via Fine-Tuned Vision-Language Models and Enhanced Reasoning Chainscited by 14
  26. 2025GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinkingcited by 14
  27. 2025MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGIcited by 14
  28. 2024LATTE: Learning to Think with Vision Specialistscited by 13
  29. 2025EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Frameworkcited by 13
  30. 2025MPBench: A Comprehensive Multimodal Reasoning Benchmark for Process Errors Identificationcited by 12
  31. 2025A Survey on Mathematical Reasoning and Optimization with Large Language Modelscited by 12
  32. 2025Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospectscited by 12
  33. 2025ViC-Bench: Benchmarking Visual-Interleaved Chain-of-Thought Capability in MLLMs with Free-Style Intermediate State Representationscited by 11
  34. 2025Corvid: Improving Multimodal Large Language Models Towards Chain-of-Thought Reasoningcited by 11
  35. 2025Benchmarking Multimodal CoT Reward Model Stepwise by Visual Programcited by 10
  36. 2025S-Chain: Structured Visual Chain-of-Thought For Medicinecited by 10
  37. 2025GenRecal: Generation after Recalibration from Large to Small Vision-Language Modelscited by 9
  38. 2025BLINK-Twice: You see, but do you observe? A Reasoning Benchmark on Visual Perceptioncited by 9

50 most cited of 120 All 120 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Models
  2. 2025MM-CoT:A Benchmark for Probing Visual Chain-of-Thought Reasoning in Multimodal Models
  3. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning
  4. 2025VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning
  5. 2026Learning Modal-Mixed Chain-of-Thought Reasoning with Latent Embeddings
  6. 2026CoLT: Teaching Multi-Modal Models to Think with Chain of Latent Thoughts
  7. 2025Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Vision
  8. 2025Rationale-Enhanced Decoding for Multi-modal Chain-of-Thought