mlsanityData to 7 Oct 2026

CMMLU: Measuring massive multitask language understanding in Chinese

arXiv 2306.09212 ↗First posted 15 Jun 2023cs.CLcited by 436

Not on the map. Why?

Cited by 51 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

As the capabilities of large language models (LLMs) continue to advance, evaluating their performance becomes increasingly crucial and challenging. This paper aims to bridge this gap by introducing CMMLU, a comprehensive Chinese benchmark that covers various subjects, including natural science, social sciences, engineering, and humanities. We conduct a thorough evaluation of 18 advanced multilingual- and Chinese-oriented LLMs, assessing their performance across different subjects and settings. The results reveal that most existing LLMs struggle to achieve an average accuracy of 50%, even when provided with in-context examples and chain-of-thought prompts, whereas the random baseline stands at 25%. This highlights significant room for improvement in LLMs. Additionally, we conduct extensive experiments to identify factors impacting the models' performance and propose directions for enhancing LLMs. CMMLU fills the gap in evaluating the knowledge and reasoning capabilities of large language models within the Chinese context.

Key facts. Cited by 436 papers in the map (data through 7 Oct 2026). Builds on 33 of its 50 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

33 of 50 references.

References · 33

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
21 more
  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  6. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  7. 2015Teaching Machines to Read and Comprehendcited by 1,039
  8. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  9. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  10. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  11. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  12. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  13. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  14. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  15. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 277
  16. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  17. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  18. 2023Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptationcited by 79
  19. 2021Understanding by Understanding Not: Modeling Negation in Language Modelscited by 73
  20. 2023M3KE: A Massive Multi-Level Multi-Subject Knowledge Evaluation Benchmark for Chinese Large Language Modelscited by 30
  21. 2023Measuring Massive Multitask Chinese Understandingcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 36; 2024: 146; 2025: 161; 2026: 93.

436 papers.

Sort:

By area:

  1. 2023Qwen Technical Reportcited by 3,273
  2. 2024Qwen2 Technical Reportcited by 2,076
  3. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  5. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  6. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  7. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  8. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  9. 2025Large Language Diffusion Modelscited by 872
  10. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  11. 2024DeepSeek-V3 Technical Reportcited by 724
  12. 2024Yi: Open Foundation Models by 01.AIcited by 706
38 more
  1. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  2. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  3. 2025Kimi K2: Open Agentic Intelligencecited by 523
  4. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  5. 2025Muon is Scalable for LLM Trainingcited by 381
  6. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  7. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  8. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 257
  9. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
  10. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  11. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  12. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  13. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  14. 2024Large Language Models for Education: A Survey and Outlookcited by 185
  15. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  16. 2026MiMo-V2-Flash Technical Reportcited by 163
  17. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 154
  18. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 127
  19. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 124
  20. 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttrainingcited by 117
  21. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 111
  22. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  23. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 102
  24. 2023SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoningcited by 94
  25. 2023HuatuoGPT-II, One-stage Training for Medical Adaption of LLMscited by 91
  26. 2024KMMLU: Measuring Massive Multitask Language Understanding in Koreancited by 91
  27. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  28. 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencentcited by 89
  29. 2024ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabiccited by 87
  30. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 84
  31. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  32. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 78
  33. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 78
  34. 2024Parameter Competition Balancing for Model Mergingcited by 71
  35. 2024OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AIcited by 70
  36. 2024TableGPT2: A Large Multimodal Model with Tabular Data Integrationcited by 69
  37. 2023Style Over Substance: Evaluation Biases for Large Language Modelscited by 68
  38. 2023Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Modelscited by 68

50 most cited of 436 All 436 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023ArcMMLU: A Library and Information Science Benchmark for Large Language Models
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
  3. 2024CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning
  4. 2025MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation
  5. 2025MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models
  6. 2024M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models
  7. 2025LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Models
  8. 2026Comprehensive Evaluation of Large Language Model Responses: A Multi-Factor Scoring System