# M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought

arXiv: https://arxiv.org/abs/2405.16473 · HTML: https://mlsanity.com/paper/2405.16473/

## Key facts

- Cited by 121 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 31 of its 50 references that are in the map.
- First posted on arXiv 26 May 2024.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://mlsanity.com/paper/2303.16199/) (2023)
- [A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge](https://mlsanity.com/paper/2206.01718/) (2022)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning](https://mlsanity.com/paper/2105.04165/) (2021)
- [Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning](https://mlsanity.com/paper/2209.14610/) (2022)
- [ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning](https://mlsanity.com/paper/2212.07919/) (2022)
- [Compositional Chain-of-Thought Prompting for Large Multimodal Models](https://mlsanity.com/paper/2311.17076/) (2023)
- [DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models](https://mlsanity.com/paper/2310.16436/) (2023)
- [Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages](https://mlsanity.com/paper/2310.14799/) (2023)
- [T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answering](https://mlsanity.com/paper/2305.03453/) (2023)
- [Abstract Visual Reasoning with Tangram Shapes](https://mlsanity.com/paper/2211.16492/) (2022)
- [The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning](https://mlsanity.com/paper/2202.04800/) (2022)
- [The Role of Chain-of-Thought in Complex Vision-Language Reasoning Task](https://mlsanity.com/paper/2311.09193/) (2023)
- [Multi-modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2312.08762/) (2023)
- [Boosting the Power of Small Multimodal Reasoning Models to Match Larger Models with Self-Consistency Training](https://mlsanity.com/paper/2311.14109/) (2023)
- [Assessing GPT4-V on Structured Reasoning Tasks](https://mlsanity.com/paper/2312.11524/) (2023)

## Built on by (most cited first, 30 of 121)

- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [From System 1 to System 2: A Survey of Reasoning Large Language Models](https://mlsanity.com/paper/2502.17419/) (2025)
- [Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization](https://mlsanity.com/paper/2411.10442/) (2024)
- [Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey](https://mlsanity.com/paper/2503.12605/) (2025)
- [PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding](https://mlsanity.com/paper/2501.16411/) (2025)
- [VisualPRM: An Effective Process Reward Model for Multimodal Reasoning](https://mlsanity.com/paper/2503.10291/) (2025)
- [MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency](https://mlsanity.com/paper/2502.09621/) (2025)
- [WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agent](https://mlsanity.com/paper/2508.05748/) (2025)
- [Visual Planning: Let's Think Only with Images](https://mlsanity.com/paper/2505.11409/) (2025)
- [Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Model](https://mlsanity.com/paper/2501.07246/) (2025)
- [OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs](https://mlsanity.com/paper/2510.10689/) (2025)
- [Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought](https://mlsanity.com/paper/2410.05695/) (2024)
- [A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges](https://mlsanity.com/paper/2412.11936/) (2024)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [Interleaved-Modal Chain-of-Thought](https://mlsanity.com/paper/2411.19488/) (2024)
- [A Survey on Evaluation of Multimodal Large Language Models](https://mlsanity.com/paper/2408.15769/) (2024)
- [VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoning](https://mlsanity.com/paper/2504.07956/) (2025)
- [Advancing Multimodal Reasoning via Reinforcement Learning with Cold Start](https://mlsanity.com/paper/2505.22334/) (2025)
- [Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought](https://mlsanity.com/paper/2505.15510/) (2025)
- [Diving into Self-Evolving Training for Multimodal Reasoning](https://mlsanity.com/paper/2412.17451/) (2024)
- [What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration](https://mlsanity.com/paper/2410.20482/) (2024)
- [CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models](https://mlsanity.com/paper/2412.12932/) (2024)
- [Explain Before You Answer: A Survey on Compositional Visual Reasoning](https://mlsanity.com/paper/2508.17298/) (2025)
- [More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models](https://mlsanity.com/paper/2509.25848/) (2025)
- [A Survey on Agentic Multimodal Large Language Models](https://mlsanity.com/paper/2510.10991/) (2025)
- [Mind with Eyes: from Language Reasoning to Multimodal Reasoning](https://mlsanity.com/paper/2503.18071/) (2025)
- [Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark](https://mlsanity.com/paper/2506.04280/) (2025)
- [HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Tasks](https://mlsanity.com/paper/2410.12381/) (2024)
- [AutoCAP: Towards Automatic Cross-lingual Alignment Planning for Zero-shot Chain-of-Thought](https://mlsanity.com/paper/2406.13940/) (2024)
- [Enhancing Advanced Visual Reasoning Ability of Large Language Models](https://mlsanity.com/paper/2409.13980/) (2024)

## Abstract (arXiv)

Multi-modal Chain-of-Thought (MCoT) requires models to leverage knowledge from both textual and visual modalities for step-by-step reasoning, which gains increasing attention. Nevertheless, the current MCoT benchmark still faces some challenges: (1) absence of visual modal reasoning, (2) single-step visual modal reasoning, and (3) Domain missing, thereby hindering the development of MCoT. Motivated by this, we introduce a novel benchmark (M3CoT) to address the above challenges, advancing the multi-domain, multi-step, and multi-modal CoT. Additionally, we conduct a thorough evaluation involving abundant MCoT approaches on Vision Large Language Models (VLLMs). In addition, we highlight that the current VLLMs still struggle to correctly reason in M3CoT and there remains a large gap between existing VLLMs and human performance in M3CoT, despite their superior results on previous MCoT benchmarks. To our knowledge, we take the first meaningful step toward the multi-domain, multi-step, and multi-modal scenario in MCoT. We hope that M3CoT can serve as a valuable resource, providing a pioneering foundation in multi-domain, multi-step, multi-modal chain-of-thought research.