mlsanityData to 7 Oct 2026

MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark

arXiv 2409.02813 ↗First posted 4 Sep 2024cs.CL cs.CVcited by 494

Not on the map. Why?

Cited by 261 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper introduces MMMU-Pro, a robust version of the Massive Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark. MMMU-Pro rigorously assesses multimodal models' true understanding and reasoning capabilities through a three-step process based on MMMU: (1) filtering out questions answerable by text-only models, (2) augmenting candidate options, and (3) introducing a vision-only input setting where questions are embedded within images. This setting challenges AI to truly "see" and "read" simultaneously, testing a fundamental human cognitive skill of seamlessly integrating visual and textual information. Results show that model performance is substantially lower on MMMU-Pro than on MMMU, ranging from 16.8% to 26.9% across models. We explore the impact of OCR prompts and Chain of Thought (CoT) reasoning, finding that OCR prompts have minimal effect while CoT generally improves performance. MMMU-Pro provides a more rigorous evaluation tool, closely mimicking real-world scenarios and offering valuable directions for future research in multimodal AI.

Key facts. Cited by 494 papers in the map (data through 7 Oct 2026). Builds on 56 of its 67 references that are in the map. First posted on arXiv 4 Sep 2024.

Builds on

56 of 67 references.

References · 56

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2015VQA: Visual Question Answeringcited by 2,489
38 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  3. 2024Qwen2 Technical Reportcited by 2,076
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  6. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  7. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  9. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  10. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  11. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  12. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  13. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  14. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  15. 2023A Survey on Multimodal Large Language Modelscited by 898
  16. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  17. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  18. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  19. 2024Yi: Open Foundation Models by 01.AIcited by 706
  20. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  21. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  22. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  23. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  24. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  25. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  26. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  27. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  28. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  29. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  30. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  31. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  32. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  33. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  34. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  35. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  36. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  37. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  38. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 29; 2025: 221; 2026: 244.

494 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2025Qwen3-VL Technical Reportcited by 2,534
  3. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  4. 2025Qwen2.5-Omni Technical Reportcited by 704
  5. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  6. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  7. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  8. 2025Qwen3-Omni Technical Reportcited by 451
  9. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  10. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  11. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  12. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 233
38 more
  1. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 212
  2. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  3. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  4. 2026Gemma 4 Technical Reportcited by 187
  5. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
  6. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  7. 2026Qwen3.5-Omni Technical Reportcited by 146
  8. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  9. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 132
  10. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 131
  11. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
  12. 2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cyclescited by 122
  13. 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agentcited by 114
  14. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 112
  15. 2025MiMo-VL Technical Reportcited by 106
  16. 2025Ovis2.5 Technical Reportcited by 103
  17. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 101
  18. 2026Kimi K3: Open Frontier Intelligencecited by 99
  19. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 96
  20. 2025Perception-Aware Policy Optimization for Multimodal Reasoningcited by 95
  21. 2025Self-Rewarding Vision-Language Model via Reasoning Decompositioncited by 95
  22. 2025Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learningcited by 86
  23. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  24. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 79
  25. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  26. 2025Wait, We Don't Need to "Wait"! Removing Thinking Tokens Improves Reasoning Efficiencycited by 65
  27. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 63
  28. 2025Scaling Language-Free Visual Representation Learningcited by 62
  29. 2025SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Modelscited by 61
  30. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Timecited by 57
  31. 2025Magistralcited by 57
  32. 2026MIRAGE: The Illusion of Visual Understandingcited by 57
  33. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  34. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 54
  35. 2025VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledgecited by 52
  36. 2025Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learningcited by 51
  37. 2026EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experiencecited by 51
  38. 2026Scaling Text-to-Image Diffusion Transformers with Representation Autoencoderscited by 49

50 most cited of 494 All 494 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation
  2. 2026When Prompts Become Pixels: Prompt-Region Grounding for Multimodal Reasoning
  3. 2025VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark
  4. 2025MM-IQ: Benchmarking Human-Like Abstraction and Reasoning in Multimodal Models
  5. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models
  6. 2025Seeing and Reasoning with Confidence: Supercharging Multimodal LLMs with an Uncertainty-Aware Agentic Framework
  7. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
  8. 2025Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark