# CUGE: A Chinese Language Understanding and Generation Evaluation Benchmark

arXiv: https://arxiv.org/abs/2112.13610 · HTML: https://mlsanity.com/paper/2112.13610/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 34 references that are in the map.
- First posted on arXiv 27 Dec 2021.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020)
- [Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension](https://mlsanity.com/paper/1904.09679/) (2019)
- [LCSTS: A Large Scale Chinese Short Text Summarization Dataset](https://mlsanity.com/paper/1506.05865/) (2015)
- [NCLS: Neural Cross-Lingual Summarization](https://mlsanity.com/paper/1909.00156/) (2019)
- [Long and Diverse Text Generation with Planning-based Hierarchical Variational Model](https://mlsanity.com/paper/1908.06605/) (2019)
- [KdConv: A Chinese Multi-domain Dialogue Dataset Towards Multi-turn Knowledge-driven Conversation](https://mlsanity.com/paper/2004.04100/) (2020)
- [CPM-2: Large-scale Cost-effective Pre-trained Language Models](https://mlsanity.com/paper/2106.10715/) (2021)
- [GLGE: A New General Language Generation Evaluation Benchmark](https://mlsanity.com/paper/2011.11928/) (2020)
- [Multi-channel Reverse Dictionary Model](https://mlsanity.com/paper/1912.08441/) (2019)
- [CCPM: A Chinese Classical Poetry Matching Dataset](https://mlsanity.com/paper/2106.01979/) (2021)

## Built on by (most cited first, 16 of 16)

- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [Understanding Emergent Abilities of Language Models from the Loss Perspective](https://mlsanity.com/paper/2403.15796/) (2024)
- [Chinese Open Instruction Generalist: A Preliminary Release](https://mlsanity.com/paper/2304.07987/) (2023)
- [Dolphin: A Challenging and Diverse Benchmark for Arabic NLG](https://mlsanity.com/paper/2305.14989/) (2023)
- [This is the way: designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polish](https://mlsanity.com/paper/2211.13112/) (2022)
- [WYWEB: A NLP Evaluation Benchmark For Classical Chinese](https://mlsanity.com/paper/2305.14150/) (2023)
- [CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models](https://mlsanity.com/paper/2402.13109/) (2024)
- [Where Are We? Evaluating LLM Performance on African Languages](https://mlsanity.com/paper/2502.19582/) (2025)
- [MuLD: The Multitask Long Document Benchmark](https://mlsanity.com/paper/2202.07362/) (2022)
- [LexSumm and LexT5: Benchmarking and Modeling Legal Summarization Tasks in English](https://mlsanity.com/paper/2410.09527/) (2024)
- [ZhuJiu: A Multi-dimensional, Multi-faceted Chinese Benchmark for Large Language Models](https://mlsanity.com/paper/2308.14353/) (2023)
- [C3Bench: A Comprehensive Classical Chinese Understanding Benchmark for Large Language Models](https://mlsanity.com/paper/2405.17732/) (2024)
- [Survey of NLU Benchmarks Diagnosing Linguistic Phenomena: Why not Standardize Diagnostics Benchmarks?](https://mlsanity.com/paper/2507.20419/) (2025)
- [InteChar: A Unified Oracle Bone Character List for Ancient Chinese Language Modeling](https://mlsanity.com/paper/2508.15791/) (2025)
- [AncientBench: Towards Comprehensive Evaluation on Excavated and Transmitted Chinese Corpora](https://mlsanity.com/paper/2512.17756/) (2025)
- [MoodBench 1.0: An Evaluation Benchmark for Emotional Companionship Dialogue Systems](https://mlsanity.com/paper/2511.18926/) (2025)

## Abstract (arXiv)

Realizing general-purpose language intelligence has been a longstanding goal for natural language processing, where standard evaluation benchmarks play a fundamental and guiding role. We argue that for general-purpose language intelligence evaluation, the benchmark itself needs to be comprehensive and systematic. To this end, we propose CUGE, a Chinese Language Understanding and Generation Evaluation benchmark with the following features: (1) Hierarchical benchmark framework, where datasets are principally selected and organized with a language capability-task-dataset hierarchy. (2) Multi-level scoring strategy, where different levels of model performance are provided based on the hierarchical framework. To facilitate CUGE, we provide a public leaderboard that can be customized to support flexible model judging criteria. Evaluation results on representative pre-trained language models indicate ample room for improvement towards general-purpose language intelligence. CUGE is publicly available at cuge.baai.ac.cn.