mlsanityData to 7 Oct 2026

MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

arXiv 2311.16502 ↗First posted 27 Nov 2023cs.CL cs.AI cs.CVcited by 2,044

Not on the map. Why?

Cited by 1,207 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce MMMU: a new benchmark designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning. MMMU includes 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering. These questions span 30 subjects and 183 subfields, comprising 30 highly heterogeneous image types, such as charts, diagrams, maps, tables, music sheets, and chemical structures. Unlike existing benchmarks, MMMU focuses on advanced perception and reasoning with domain-specific knowledge, challenging models to perform tasks akin to those faced by experts. The evaluation of 14 open-source LMMs as well as the proprietary GPT-4V(ision) and Gemini highlights the substantial challenges posed by MMMU. Even the advanced GPT-4V and Gemini Ultra only achieve accuracies of 56% and 59% respectively, indicating significant room for improvement. We believe MMMU will stimulate the community to build next-generation multimodal foundation models towards expert artificial general intelligence.

Key facts. Cited by 2,044 papers in the map (data through 7 Oct 2026). Builds on 55 of its 84 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

55 of 84 references.

References · 55

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  13. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  14. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  15. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  16. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  17. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  18. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  19. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  20. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  21. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  22. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  23. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  24. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  25. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  26. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  27. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  28. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  29. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  30. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  31. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  32. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  33. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  34. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  35. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  36. 2023OpenAGI: When LLM Meets Domain Expertscited by 196
  37. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  38. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 475; 2025: 834; 2026: 725.

2,044 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  3. 2025Qwen3-VL Technical Reportcited by 2,534
  4. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  5. 2025Gemma 3 Technical Reportcited by 1,686
  6. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  7. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  8. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  9. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  10. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  11. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  12. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
38 more
  1. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  2. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  3. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  4. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  5. 2025Qwen2.5-Omni Technical Reportcited by 704
  6. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  7. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  8. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  9. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  10. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  11. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  12. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  13. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  14. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  15. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  16. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 476
  17. 2025Qwen3-Omni Technical Reportcited by 451
  18. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  19. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  20. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  21. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  22. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  23. 2025Kimi-VL Technical Reportcited by 381
  24. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  25. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  26. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  27. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  28. 2024What matters when building vision-language models?cited by 321
  29. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  30. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  31. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  32. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  33. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  34. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  35. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  36. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  37. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  38. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 267

50 most cited of 2,044 All 2,044 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning
  2. 2025MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models
  3. 2023Enhancing Human-like Multi-Modal Reasoning: A New Challenging Dataset and Comprehensive Framework
  4. 2026MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos
  5. 2024M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models
  6. 2025CSVQA: A Chinese Multimodal Benchmark for Evaluating STEM Reasoning Capabilities of VLMs
  7. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
  8. 2025VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark