mlsanityData to 14 Oct 2026

CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning

arXiv 2401.14011 ↗First posted 25 Jan 2024cs.CL cs.AI cs.MMcited by 31

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 1 more
Abstract (arXiv)

Multi-modal large language models(MLLMs) have achieved remarkable progress and demonstrated powerful knowledge comprehension and reasoning abilities. However, the mastery of domain-specific knowledge, which is essential for evaluating the intelligence of MLLMs, continues to be a challenge. Current multi-modal benchmarks for domain-specific knowledge concentrate on multiple-choice questions and are predominantly available in English, which imposes limitations on the comprehensiveness of the evaluation. To this end, we introduce CMMU, a novel benchmark for multi-modal and multi-type question understanding and reasoning in Chinese. CMMU consists of 3,603 questions in 7 subjects, covering knowledge from primary to high school. The questions can be categorized into 3 types: multiple-choice, multiple-response, and fill-in-the-blank, bringing greater challenges to MLLMs. In addition, we propose an evaluation strategy called Positional Error Variance for assessing multiple-choice questions. The strategy aims to perform a quantitative analysis of position bias. We evaluate seven open-source MLLMs along with GPT4-V, Gemini-Pro, and Qwen-VL-Plus. The results demonstrate that CMMU poses a significant challenge to the recent MLLMs. The data and code are available at https://github.com/FlagOpen/CMMU.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 30 of its 36 references that are in the map. First posted on arXiv 25 Jan 2024.

Builds on

30 of 36 references.

References · 30

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  11. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
18 more
  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  3. 2019Towards VQA Models That Can Readcited by 1,718
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  6. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  7. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  8. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  9. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  10. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  11. 2018nocaps: novel object captioning at scalecited by 444
  12. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  13. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  14. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 110
  15. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 97
  16. 2017An Analysis of Visual Question Answering Algorithmscited by 90
  17. 2023BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Modelscited by 41
  18. 2023CLEVA: Chinese Language Models EVAluation Platformcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 18; 2026: 4.

31 papers.

Sort:

By area:

  1. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  2. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 82
  3. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  4. 2024ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Areacited by 71
  5. 2025RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?cited by 60
  6. 2024ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detectioncited by 37
  7. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  8. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  9. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  10. 2024Can MLLMs Understand the Deep Implication Behind Chinese Images?cited by 14
  11. 2025UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Modelscited by 13
  12. 2025BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Datasetcited by 12
19 more
  1. 2025Reinforcement Learning for Large Model: A Surveycited by 9
  2. 2024MULTI: Multimodal Understanding Leaderboard with Text and Imagescited by 7
  3. 2025K12Vista: Exploring the Boundaries of MLLMs in K-12 Educationcited by 6
  4. 2024MathScape: Benchmarking Multimodal Large Language Models in Real-World Mathematical Contextscited by 5
  5. 2025REVAL: A Comprehension Evaluation on Reliability and Values of Large Vision-Language Modelscited by 5
  6. 2025InsightVision: A Comprehensive, Multi-Level Chinese-based Benchmark for Evaluating Implicit Visual Semantics in Large Vision Language Modelscited by 4
  7. 2025AutoJudger: An Agent-Driven Framework for Efficient Benchmarking of MLLMscited by 4
  8. 2025VisTW: Benchmarking Vision-Language Models for Traditional Chinese in Taiwancited by 3
  9. 2025Abstractive Visual Understanding of Multi-modal Structured Knowledge: A New Perspective for MLLM Evaluationcited by 3
  10. 2025FlagEvalMM: A Flexible Framework for Comprehensive Multimodal Model Evaluationcited by 3
  11. 2025MELAC: Massive Evaluation of Large Language Models with Alignment of Culture in Persian Languagecited by 3
  12. 2025EduFlow: Advancing MLLMs' Problem-Solving Proficiency through Multi-Stage, Multi-Perspective Critiquecited by 2
  13. 2025MACEval: A Multi-Agent Continual Evaluation Network for Large Modelscited by 2
  14. 2026SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integrationcited by 2
  15. 2025Argus Inspection: Do Multimodal Large Language Models Possess the Eye of Panoptes?cited by 1
  16. 2026CArtBench: Evaluating Vision-Language Models on Chinese Art Understanding, Interpretation, and Authenticitycited by 1
  17. 2025SparseDoctor: Towards Efficient Chat Doctor with Mixture of Experts Enhanced Large Language Modelscited by 0
  18. 2026Benchmarking Large Vision-Language Models on CFMME: A Comprehensive Chinese Financial Multimodal Evaluation Datasetcited by 0
  19. 2026OmniEdu: Open Foundation Models for Learning and Teachingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023CMMLU: Measuring massive multitask language understanding in Chinese
  2. 2025MMLU-Reason: Benchmarking Multi-Task Multi-modal Language Understanding and Reasoning
  3. 2025OmniEduBench: A Comprehensive Chinese Benchmark for Evaluating Large Language Models in Education
  4. 2025MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models
  5. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
  6. 2025Reasoning Models are Test Exploiters: Rethinking Multiple-Choice
  7. 2025Benchmarking Chinese Commonsense Reasoning with a Multi-hop Reasoning Perspective
  8. 2025MDK12-Bench: A Comprehensive Evaluation of Multimodal Large Language Models on Multidisciplinary Exams