# WYWEB: A NLP Evaluation Benchmark For Classical Chinese

arXiv: https://arxiv.org/abs/2305.14150 · HTML: https://mlsanity.com/paper/2305.14150/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 51 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020)
- [Revisiting Pre-Trained Models for Chinese Natural Language Processing](https://mlsanity.com/paper/2004.13922/) (2020)
- [StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding](https://mlsanity.com/paper/1908.04577/) (2019)
- [CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2106.08087/) (2021)
- [A Span-Extraction Dataset for Chinese Machine Reading Comprehension](https://mlsanity.com/paper/1810.07366/) (2018)
- [ChID: A Large-scale Chinese IDiom Dataset for Cloze Test](https://mlsanity.com/paper/1906.01265/) (2019)
- [The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English](https://mlsanity.com/paper/1902.01382/) (2019)
- [CLiMP: A Benchmark for Chinese Language Model Evaluation](https://mlsanity.com/paper/2101.11131/) (2021)
- [NEZHA: Neural Contextualized Representation for Chinese Language Understanding](https://mlsanity.com/paper/1909.00204/) (2019)
- [CJRC: A Reliable Human-Annotated Benchmark DataSet for Chinese Judicial Reading Comprehension](https://mlsanity.com/paper/1912.09156/) (2019)
- [CUGE: A Chinese Language Understanding and Generation Evaluation Benchmark](https://mlsanity.com/paper/2112.13610/) (2021)
- [CCPM: A Chinese Classical Poetry Matching Dataset](https://mlsanity.com/paper/2106.01979/) (2021)
- [A Sentence Cloze Dataset for Chinese Machine Reading Comprehension](https://mlsanity.com/paper/2004.03116/) (2020)
- [Time-Aware Ancient Chinese Text Translation and Inference](https://mlsanity.com/paper/2107.03179/) (2021)
- [QuoteR: A Benchmark of Quote Recommendation for Writing](https://mlsanity.com/paper/2202.13145/) (2022)
- [Corpus of Chinese Dynastic Histories: Gender Analysis over Two Millennia](https://mlsanity.com/paper/2005.08793/) (2020)

## Built on by (most cited first, 12 of 12)

- [AC-EVAL: Evaluating Ancient Chinese Language Understanding in Large Language Models](https://mlsanity.com/paper/2403.06574/) (2024)
- [F`ux`i: A Benchmark for Evaluating Language Models on Ancient Chinese Text Understanding and Generation](https://mlsanity.com/paper/2503.15837/) (2025)
- [Surveying the Dead Minds: Historical-Psychological Text Analysis with Contextualized Construct Representation (CCR) for Classical Chinese](https://mlsanity.com/paper/2403.00509/) (2024)
- [Benchmarking Machine Translation on Chinese Social Media Texts](https://mlsanity.com/paper/2601.22931/) (2026)
- [LFED: A Literary Fiction Evaluation Dataset for Large Language Models](https://mlsanity.com/paper/2405.10166/) (2024)
- [Shared Heritage, Distinct Writing: Rethinking Resource Selection for East Asian Historical Documents](https://mlsanity.com/paper/2411.04822/) (2024)
- [System Report for CCL25-Eval Task 5: New Dataset and LoRA-Fine-Tuned Qwen2.5](https://mlsanity.com/paper/2606.12392/) (2026)
- [Translation via Annotation: A Computational Study of Translating Classical Chinese into Japanese](https://mlsanity.com/paper/2511.05239/) (2025)
- [MCGA: A Multi-task Classical Chinese Literary Genre Audio Corpus](https://mlsanity.com/paper/2601.09270/) (2026)
- [Do Evaluation Metrics Detect Errors in Classical Chinese to English Translations?](https://mlsanity.com/paper/2608.08283/) (2026)
- [L"ushi-1840: Building and Probing a Knowledge Cutoff for a Classical Chinese Language Model](https://mlsanity.com/paper/2610.16201/) (2026)
- [Does the Linguistic Register of Thinking Matter? Wenyan as a Compact Reasoning Register for Chinese LLMs](https://mlsanity.com/paper/2610.19741/) (2026)

## Abstract (arXiv)

To fully evaluate the overall performance of different NLP models in a given domain, many evaluation benchmarks are proposed, such as GLUE, SuperGLUE and CLUE. The fi eld of natural language understanding has traditionally focused on benchmarks for various tasks in languages such as Chinese, English, and multilingua, however, there has been a lack of attention given to the area of classical Chinese, also known as "wen yan wen", which has a rich history spanning thousands of years and holds signifi cant cultural and academic value. For the prosperity of the NLP community, in this paper, we introduce the WYWEB evaluation benchmark, which consists of nine NLP tasks in classical Chinese, implementing sentence classifi cation, sequence labeling, reading comprehension, and machine translation. We evaluate the existing pre-trained language models, which are all struggling with this benchmark. We also introduce a number of supplementary datasets and additional tools to help facilitate further progress on classical Chinese NLU. The github repository is https://github.com/baudzhou/WYWEB.