mlsanityData to 7 Oct 2026

CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark

arXiv 2401.11944 ↗First posted 22 Jan 2024cs.CL cs.AI cs.CVcited by 54

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

As the capabilities of large multimodal models (LMMs) continue to advance, evaluating the performance of LMMs emerges as an increasing need. Additionally, there is an even larger gap in evaluating the advanced knowledge and reasoning abilities of LMMs in non-English contexts such as Chinese. We introduce CMMMU, a new Chinese Massive Multi-discipline Multimodal Understanding benchmark designed to evaluate LMMs on tasks demanding college-level subject knowledge and deliberate reasoning in a Chinese context. CMMMU is inspired by and strictly follows the annotation and analysis pattern of MMMU. CMMMU includes 12k manually collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering, like its companion, MMMU. These questions span 30 subjects and comprise 39 highly heterogeneous image types, such as charts, diagrams, maps, tables, music sheets, and chemical structures. CMMMU focuses on complex perception and reasoning with domain-specific knowledge in the Chinese context. We evaluate 11 open-source LLMs and one proprietary GPT-4V(ision). Even GPT-4V only achieves accuracies of 42%, indicating a large space for improvement. CMMMU will boost the community to build the next-generation LMMs towards expert artificial intelligence and promote the democratization of LMMs by providing diverse language contexts.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 34 references that are in the map. First posted on arXiv 22 Jan 2024.

Builds on

31 of 34 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  2. 2023Generative Multimodal Models are In-Context Learners20 Dec
  3. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  4. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages23 Aug
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  6. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 23

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2023Qwen Technical Reportcited by 3,273
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  11. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  12. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
11 more
  1. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  2. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  3. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  4. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  5. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  6. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  7. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  8. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  9. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 182
  10. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  11. 2023Towards Models that Can See and Readcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 28; 2025: 20; 2026: 6.

54 papers.

Sort:

By area:

  1. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  2. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 476
  3. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  4. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  5. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  6. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  7. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 100
  8. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  9. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  10. 2024OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AIcited by 70
  11. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
  12. 2025SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Modelscited by 61
38 more
  1. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Textcited by 58
  2. 2025Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learningcited by 51
  3. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  4. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 43
  5. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  6. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  7. 2024ICONS: Influence Consensus for Vision-Language Data Selectioncited by 28
  8. 2025MM-IQ: Benchmarking Human-Like Abstraction and Reasoning in Multimodal Modelscited by 28
  9. 2024M5 -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Taskscited by 27
  10. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  11. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  12. 2024Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Modelscited by 24
  13. 2024Training on the Benchmark Is Not All You Needcited by 24
  14. 2025VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videoscited by 24
  15. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluationcited by 17
  16. 2025HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Modelscited by 17
  17. 2024CMMaTH: A Chinese Multi-modal Math Skill Evaluation Benchmark for Foundation Modelscited by 15
  18. 2024Multi-label Cluster Discrimination for Visual Representation Learningcited by 14
  19. 2024VisScience: An Extensive Benchmark for Evaluating K12 Educational Multi-modal Scientific Reasoningcited by 13
  20. 2025UNO-Bench: A Unified Benchmark for Exploring the Compositional Law Between Uni-modal and Omni-modal in Omni Modelscited by 12
  21. 2024ING-VP: MLLMs cannot Play Easy Vision-based Games Yetcited by 10
  22. 2025Reinforcement Learning for Large Model: A Surveycited by 8
  23. 2024MULTI: Multimodal Understanding Leaderboard with Text and Imagescited by 7
  24. 2024LIME: Less Is More for MLLM Evaluationcited by 7
  25. 2024PIN: A Knowledge-Intensive Dataset for Paired and Interleaved Multimodal Documentscited by 6
  26. 2024Can We Predict Performance of Large Models across Vision-Language Tasks?cited by 6
  27. 2025IV-Bench: A Benchmark for Image-Grounded Video Perception and Reasoning in Multimodal LLMscited by 6
  28. 2025REVAL: A Comprehension Evaluation on Reliability and Values of Large Vision-Language Modelscited by 5
  29. 2025MathOPEval: A Fine-grained Evaluation Benchmark for Visual Operations of MLLMs in Mathematical Reasoningcited by 4
  30. 2025AutoJudger: An Agent-Driven Framework for Efficient Benchmarking of MLLMscited by 3
  31. 2025Learning What Matters: Prioritized Concept Learning via Relative Error-driven Sample Selectioncited by 3
  32. 2025Toward Real-World Table Agents: Capabilities, Workflows, and Design Principles for LLM-based Table Intelligencecited by 2
  33. 2025AgriEval: A Comprehensive Chinese Agricultural Benchmark for Large Language Modelscited by 2
  34. 2026From UAV Imagery to Agronomic Reasoning: A Multimodal LLM Benchmark for Plant Phenotypingcited by 1
  35. 2025VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmarkcited by 0
  36. 2025The Percept-V Challenge: Can Multimodal LLMs Crack Simple Perception Problems?cited by 0
  37. 2025ACADREASON: Exploring the Limits of Reasoning Models with Academic Research Problemscited by 0
  38. 2026DanQing: An Up-to-Date Large-Scale Chinese Vision-Language Pre-training Datasetcited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset
  2. 2025MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problems
  3. 2024M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models
  4. 2023CMMLU: Measuring massive multitask language understanding in Chinese
  5. 2024GAOKAO-MM: A Chinese Human-Level Benchmark for Multimodal Models Evaluation
  6. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark
  7. 2024CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning
  8. 2025Multi-Physics: A Comprehensive Benchmark for Multimodal LLMs Reasoning on Chinese Multi-Subject Physics Problems