mlsanityData to 13 Oct 2026

MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency

arXiv 2502.09621 ↗First posted 13 Feb 2025cs.CV cs.AI cs.CLcited by 125

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Answering questions with Chain-of-Thought (CoT) has significantly enhanced the reasoning capabilities of Large Language Models (LLMs), yet its impact on Large Multimodal Models (LMMs) still lacks a systematic assessment and in-depth investigation. In this paper, we introduce MME-CoT, a specialized benchmark evaluating the CoT reasoning performance of LMMs, spanning six domains: math, science, OCR, logic, space-time, and general scenes. As the first comprehensive study in this area, we propose a thorough evaluation suite incorporating three novel metrics that assess the reasoning quality, robustness, and efficiency at a fine-grained level. Leveraging curated high-quality data and a unique evaluation strategy, we conduct an in-depth analysis of state-of-the-art LMMs, uncovering several key insights: 1) Models with reflection mechanism demonstrate a superior CoT quality, with Kimi k1.5 outperforming GPT-4o and demonstrating the highest quality results; 2) CoT prompting often degrades LMM performance on perception-heavy tasks, suggesting a potentially harmful overthinking behavior; and 3) Although the CoT quality is high, LMMs with reflection exhibit significant inefficiency in both normal response and self-correction phases. We hope MME-CoT serves as a foundation for advancing multimodal reasoning in LMMs. Project Page: https://mmecot.github.io/

Key facts. Cited by 125 papers in the map (data through 13 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 40 of its 53 references that are in the map. First posted on arXiv 13 Feb 2025.

Builds on

40 of 53 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Step23 Jan
  2. 2025Virgo: A Preliminary Exploration on Reproducing o1-like MLLM3 Jan
  3. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  4. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization15 Nov
  5. 2024M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought26 May
  6. 2024How Far Are We from Intelligent Visual Deductive Reasoning?7 Mar

Other references · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  4. 2023Visual Instruction Tuningcited by 7,827
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,314
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,191
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,703
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,692
  9. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,230
  10. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,201
  11. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,070
  12. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,041
22 more
  1. 2023VideoChat: Chat-Centric Video Understandingcited by 854
  2. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 580
  3. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 576
  4. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 446
  5. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 329
  6. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 285
  7. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 258
  8. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 231
  9. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 193
  10. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  11. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 180
  12. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 176
  13. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 173
  14. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 131
  15. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  16. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  17. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 100
  18. 2023ReCEval: Evaluating Reasoning Chains via Correctness and Informativenesscited by 76
  19. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Modelscited by 70
  20. 2024Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulationcited by 46
  21. 2024SAM2Point: Segment Any 3D as Videos in Zero-shot and Promptable Mannerscited by 18
  22. 2024Chimera: Improving Generalist Model with Domain-Specific Expertscited by 11

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 83; 2026: 42.

125 papers.

Sort:

By area:

  1. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  2. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 179
  3. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 178
  4. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 163
  5. 2025Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generationcited by 98
  6. 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentationcited by 86
  7. 2025VGR: Visual Grounded Reasoningcited by 72
  8. 2025WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratchcited by 69
  9. 2025Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodologycited by 67
  10. 2025Grounded Chain-of-Thought for Multimodal Large Language Modelscited by 58
  11. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 56
  12. 2025ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoningcited by 56
38 more
  1. 2026Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understandingcited by 52
  2. 2025An Agentic System for Rare Disease Diagnosis with Traceable Reasoningcited by 50
  3. 2025Think or Not Think: A Study of Explicit Thinking in Rule-Based Visual Reinforcement Fine-Tuningcited by 44
  4. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 44
  5. 2025VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoningcited by 41
  6. 2025Are Video Models Ready as Zero-Shot Reasoners? An Empirical Study with the MME-CoF Benchmarkcited by 41
  7. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  8. 2025Jigsaw-R1: A Study of Rule-based Visual Reinforcement Learning with Jigsaw Puzzlescited by 36
  9. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 35
  10. 2025MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMscited by 32
  11. 2025Reinforcing Video Reasoning with Focused Thinkingcited by 27
  12. 2025VERIFY: A Benchmark of Visual Explanation and Reasoning for Investigating Multimodal Reasoning Fidelitycited by 23
  13. 2025SciVerse: Unveiling the Knowledge Comprehension and Visual Reasoning of LMMs on Multi-modal Scientific Problemscited by 22
  14. 2025Mind with Eyes: from Language Reasoning to Multimodal Reasoningcited by 22
  15. 2025A Survey on Agentic Multimodal Large Language Modelscited by 22
  16. 2025Reasoning via Video: The First Evaluation of Video Models' Reasoning Abilities through Maze-Solving Taskscited by 22
  17. 2025OmniCaptioner: One Captioner to Rule Them Allcited by 20
  18. 2025Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmarkcited by 20
  19. 2025THINK-Bench: Evaluating Thinking Efficiency and Chain-of-Thought Quality of Large Reasoning Modelscited by 19
  20. 2025MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLMcited by 17
  21. 2025MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Modelscited by 17
  22. 2025Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Modelscited by 14
  23. 2025GRE Suite: Geo-localization Inference via Fine-Tuned Vision-Language Models and Enhanced Reasoning Chainscited by 14
  24. 2025OmniEarth-Bench: Towards Holistic Evaluation of Earth's Six Spheres and Cross-Spheres Interactions with Multimodal Observational Earth Datacited by 14
  25. 2025MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGIcited by 14
  26. 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 14
  27. 2025TrustGeoGen: Formal-Verified Data Engine for Trustworthy Multi-modal Geometric Problem Solvingcited by 13
  28. 2025DraCo: Draft as CoT for Text-to-Image Preview and Rare Concept Generationcited by 13
  29. 2025ViC-Bench: Benchmarking Visual-Interleaved Chain-of-Thought Capability in MLLMs with Free-Style Intermediate State Representationscited by 11
  30. 2025CrossLMM: Decoupling Long Video Sequences from LMMs via Dual Cross-Attention Mechanismscited by 11
  31. 2025Understanding-in-Generation: Reinforcing Generative Capability of Unified Model via Infusing Understanding into Generationcited by 11
  32. 2025From What to Why: A Multi-Agent System for Evidence-based Chemical Reaction Condition Reasoningcited by 11
  33. 2025DualVLA: Building a Generalizable Embodied Agent via Partial Decoupling of Reasoning and Actioncited by 11
  34. 2026On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMscited by 11
  35. 2025CoRe3D: Collaborative Reasoning as a Foundation for 3D Intelligencecited by 10
  36. 2025VER-Bench: Evaluating MLLMs on Reasoning with Fine-Grained Visual Evidencecited by 9
  37. 2025BLINK-Twice: You see, but do you observe? A Reasoning Benchmark on Visual Perceptioncited by 9
  38. 2025PiSA: A Self-Augmented Data Engine and Training Strategy for 3D Understanding with Large Modelscited by 8

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning
  2. 2023Multimodal Chain-of-Thought Reasoning in Language Models
  3. 2026Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning
  4. 2025MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models
  5. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning
  6. 2025Cross Domain Evaluation of Multimodal Chain-of-Thought Reasoning of different datasets into the Amazon CoT Framework
  7. 2024ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting
  8. 2025CoT Vectors: Transferring and Probing the Reasoning Mechanisms of LLMs