mlsanityData to 7 Oct 2026

Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark

arXiv 2501.05444 ↗First posted 9 Jan 2025cs.CVcited by 132

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

The ability to organically reason over and with both text and images is a pillar of human intelligence, yet the ability of Multimodal Large Language Models (MLLMs) to perform such multimodal reasoning remains under-explored. Existing benchmarks often emphasize text-dominant reasoning or rely on shallow visual cues, failing to adequately assess integrated visual and textual reasoning. We introduce EMMA (Enhanced MultiModal reAsoning), a benchmark targeting organic multimodal reasoning across mathematics, physics, chemistry, and coding. EMMA tasks demand advanced cross-modal reasoning that cannot be addressed by reasoning independently in each modality, offering an enhanced test suite for MLLMs' reasoning capabilities. Our evaluation of state-of-the-art MLLMs on EMMA reveals significant limitations in handling complex multimodal and multi-step reasoning tasks, even with advanced techniques like Chain-of-Thought prompting and test-time compute scaling underperforming. These findings underscore the need for improved multimodal architectures and training paradigms to close the gap between human and model reasoning in multimodality.

Key facts. Cited by 132 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 47 of its 92 references that are in the map. First posted on arXiv 9 Jan 2025.

Builds on

47 of 92 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models17 Dec
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  3. 2024ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation14 Jun
  4. 2024Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Models22 May
  5. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning25 Mar
  6. 2024VISREAS: Complex Visual Reasoning with Unanswerable Questions23 Feb
  7. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning19 Feb
  8. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb
  9. 2022DePlot: One-shot visual language reasoning by plot-to-table translation20 Dec
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning19 Mar

Other references · 37

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  12. 2024Qwen2 Technical Reportcited by 2,076
25 more
  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  2. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  5. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  7. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  8. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  9. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  10. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  11. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  12. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  13. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  14. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 229
  15. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
  16. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 158
  17. 2024Marco-o1: Towards Open Reasoning Models for Open-Ended Solutionscited by 142
  18. 2019On the Measure of Intelligencecited by 129
  19. 2024Does Spatial Cognition Emerge in Frontier Models?cited by 65
  20. 2024EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Modelscited by 41
  21. 2024MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problemscited by 38
  22. 2024IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representationscited by 37
  23. 2024Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large Language Models in Code Generation from Scientific Plotscited by 20
  24. 2024Arena-Lite: Efficient and Reliable Large Language Model Evaluation via Tournament-Based Direct Comparisonscited by 7
  25. 2024SMiCRM: A Benchmark Dataset of Mechanistic Molecular Imagescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 85; 2026: 47.

132 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  3. 2025Learning to Reason under Off-Policy Guidancecited by 266
  4. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  5. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  6. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 174
  7. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
  8. 2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cyclescited by 122
  9. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 102
  10. 2025Visual Planning: Let's Think Only with Imagescited by 85
  11. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 78
  12. 2025Wait, We Don't Need to "Wait"! Removing Thinking Tokens Improves Reasoning Efficiencycited by 65
38 more
  1. 2025PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoningcited by 58
  2. 2025ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoningcited by 55
  3. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 54
  4. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 47
  5. 2025Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Rewardcited by 45
  6. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 43
  7. 2025Emergent Hierarchical Reasoning in LLMs through Reinforcement Learningcited by 42
  8. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Modelscited by 40
  9. 2025Jigsaw-R1: A Study of Rule-based Visual Reinforcement Learning with Jigsaw Puzzlescited by 36
  10. 2025LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Modelcited by 34
  11. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  12. 2025Point-RFT: Improving Multimodal Reasoning with Visually Grounded Reinforcement Finetuningcited by 31
  13. 2025MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMscited by 31
  14. 2025VGRP-Bench: Visual Grid Reasoning Puzzle Benchmark for Large Vision-Language Modelscited by 26
  15. 2025VL-Cogito: Progressive Curriculum Reinforcement Learning for Advanced Multimodal Reasoningcited by 26
  16. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  17. 2025FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflowcited by 25
  18. 2025ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMscited by 25
  19. 2025v1: Learning to Point Visual Tokens for Multimodal Grounded Reasoningcited by 24
  20. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  21. 2025Mind with Eyes: from Language Reasoning to Multimodal Reasoningcited by 22
  22. 2025ReasonMap: Towards Fine-Grained Visual Reasoning from Transit Mapscited by 20
  23. 2025SurveillanceVQA-589K: A Benchmark for Comprehensive Surveillance Video-Language Understanding with Large Modelscited by 18
  24. 2025When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thoughtcited by 18
  25. 2025TiViBench: Benchmarking Think-in-Video Reasoning for Video Generative Modelscited by 18
  26. 2025Generative Universal Verifier as Multimodal Meta-Reasonercited by 17
  27. 2025Skywork-R1V3 Technical Reportcited by 16
  28. 2025Look Again, Think Slowly: Enhancing Visual Reflection in Vision-Language Modelscited by 16
  29. 2025Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Imagecited by 16
  30. 2025Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLMcited by 14
  31. 2025AMSbench: A Comprehensive Benchmark for Evaluating MLLM Capabilities in AMS Circuitscited by 14
  32. 2025PhysUniBench: A Multi-Modal Physics Reasoning Benchmark at Undergraduate Levelcited by 14
  33. 2025Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMscited by 14
  34. 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 14
  35. 2025MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Modelscited by 13
  36. 2025Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Modelscited by 13
  37. 2025Unveiling the Compositional Ability Gap in Vision-Language Reasoning Modelcited by 13
  38. 2025ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generationcited by 13

50 most cited of 132 All 132 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning
  2. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models
  3. 2025Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark
  4. 2025MM-IQ: Benchmarking Human-Like Abstraction and Reasoning in Multimodal Models
  5. 2026MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning
  6. 2025Argus Inspection: Do Multimodal Large Language Models Possess the Eye of Panoptes?
  7. 2024Can Multimodal Large Language Model Think Analogically?
  8. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts