# Measuring Massive Multitask Chinese Understanding

arXiv: https://arxiv.org/abs/2304.12986 · HTML: https://mlsanity.com/paper/2304.12986/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 6 references that are in the map.
- First posted on arXiv 25 Apr 2023.

## Builds on (most cited first, 3 of 3)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)

## Built on by (most cited first, 25 of 25)

- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [CMMLU: Measuring massive multitask language understanding in Chinese](https://mlsanity.com/paper/2306.09212/) (2023)
- [Evaluating Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2310.19736/) (2023)
- [SafetyBench: Evaluating the Safety of Large Language Models](https://mlsanity.com/paper/2309.07045/) (2023)
- [Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Dataset](https://mlsanity.com/paper/2306.03030/) (2023)
- [CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility](https://mlsanity.com/paper/2307.09705/) (2023)
- [Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation](https://mlsanity.com/paper/2306.05783/) (2023)
- [A Comprehensive Evaluation of Quantization Strategies for Large Language Models](https://mlsanity.com/paper/2402.16775/) (2024)
- [Large Language Models for Education: A Survey](https://mlsanity.com/paper/2405.13001/) (2024)
- [DialogBench: Evaluating LLMs as Human-like Dialogue Systems](https://mlsanity.com/paper/2311.01677/) (2023)
- [M3KE: A Massive Multi-Level Multi-Subject Knowledge Evaluation Benchmark for Chinese Large Language Models](https://mlsanity.com/paper/2305.10263/) (2023)
- [An Empirical Study of Instruction-tuning Large Language Models in Chinese](https://mlsanity.com/paper/2310.07328/) (2023)
- [CLEVA: Chinese Language Models EVAluation Platform](https://mlsanity.com/paper/2308.04813/) (2023)
- [E-EVAL: A Comprehensive Chinese K-12 Education Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2401.15927/) (2024)
- [Evaluating the Generation Capabilities of Large Chinese Language Models](https://mlsanity.com/paper/2308.04823/) (2023)
- [ERNIE 5.0 Technical Report](https://mlsanity.com/paper/2602.04705/) (2026)
- [Dr.Academy: A Benchmark for Evaluating Questioning Capability in Education for Large Language Models](https://mlsanity.com/paper/2408.10947/) (2024)
- [AECBench: A Hierarchical Benchmark for Knowledge Evaluation of Large Language Models in the AEC Field](https://mlsanity.com/paper/2509.18776/) (2025)
- [ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled Tuning](https://mlsanity.com/paper/2510.10071/) (2025)
- [What is the best model? Application-driven Evaluation for Large Language Models](https://mlsanity.com/paper/2406.10307/) (2024)
- [Online Training of Large Language Models: Learn while chatting](https://mlsanity.com/paper/2403.04790/) (2024)
- [Revolutionizing Bridge Operation and Maintenance with LLM-based Agents: An Overview of Applications and Insights](https://mlsanity.com/paper/2407.10064/) (2024)
- [Identity Lock: Locking API Fine-tuned LLMs With Identity-based Wake Words](https://mlsanity.com/paper/2503.10668/) (2025)
- [EduEval: A Hierarchical Cognitive Benchmark for Evaluating Large Language Models in Chinese Education](https://mlsanity.com/paper/2512.00290/) (2025)

## Abstract (arXiv)

The development of large-scale Chinese language models is flourishing, yet there is a lack of corresponding capability assessments. Therefore, we propose a test to measure the multitask accuracy of large Chinese language models. This test encompasses four major domains, including medicine, law, psychology, and education, with 15 subtasks in medicine and 8 subtasks in education. We found that the best-performing models in the zero-shot setting outperformed the worst-performing models by nearly 18.6 percentage points on average. Across the four major domains, the highest average zero-shot accuracy of all models is 0.512. In the subdomains, only the GPT-3.5-turbo model achieved a zero-shot accuracy of 0.693 in clinical medicine, which was the highest accuracy among all models across all subtasks. All models performed poorly in the legal domain, with the highest zero-shot accuracy reaching only 0.239. By comprehensively evaluating the breadth and depth of knowledge across multiple disciplines, this test can more accurately identify the shortcomings of the models.