# Can Large Language Model Comprehend Ancient Chinese? A Preliminary Test on ACLUE

arXiv: https://arxiv.org/abs/2310.09550 · HTML: https://mlsanity.com/paper/2310.09550/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 34 references that are in the map.
- First posted on arXiv 14 Oct 2023.

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [CMMLU: Measuring massive multitask language understanding in Chinese](https://mlsanity.com/paper/2306.09212/) (2023)
- [Generating Chinese Classical Poems with RNN Encoder-Decoder](https://mlsanity.com/paper/1604.01537/) (2016)
- [AnchiBERT: A Pre-Trained Model for Ancient ChineseLanguage Understanding and Generation](https://mlsanity.com/paper/2009.11473/) (2020)
- [Ancient-Modern Chinese Translation with a Large Training Dataset](https://mlsanity.com/paper/1808.03738/) (2018)
- [Sentence Segmentation for Classical Chinese Based on LSTM with Radical Embedding](https://mlsanity.com/paper/1810.03479/) (2018)

## Built on by (most cited first, 10 of 10)

- [BayLing 2: A Multilingual Large Language Model with Efficient Language Alignment](https://mlsanity.com/paper/2411.16300/) (2024)
- [Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese](https://mlsanity.com/paper/2505.22645/) (2025)
- [F`ux`i: A Benchmark for Evaluating Language Models on Ancient Chinese Text Understanding and Generation](https://mlsanity.com/paper/2503.15837/) (2025)
- [C3Bench: A Comprehensive Classical Chinese Understanding Benchmark for Large Language Models](https://mlsanity.com/paper/2405.17732/) (2024)
- [TELEVAL: A Benchmark Designed for Spoken Language Models in Chinese Interactive Scenarios](https://mlsanity.com/paper/2507.18061/) (2025)
- [Measuring Hong Kong Massive Multi-Task Language Understanding](https://mlsanity.com/paper/2505.02177/) (2025)
- [Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models](https://mlsanity.com/paper/2409.20181/) (2024)
- [OmniEvalKit: A Modular, Lightweight Toolbox for Evaluating Large Language Model and its Omni-Extensions](https://mlsanity.com/paper/2412.06693/) (2024)
- [AI Historian: Helping historians organize and verify person-centred temporal clues from dispersed historical narratives](https://mlsanity.com/paper/2608.29133/) (2026)
- [Neo-Classic: A Benchmark for Evaluating Linguistic-Aesthetic Reasoning in Classical Chinese Poetry](https://mlsanity.com/paper/2609.19154/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have showcased remarkable capabilities in understanding and generating language. However, their ability in comprehending ancient languages, particularly ancient Chinese, remains largely unexplored. To bridge this gap, we present ACLUE, an evaluation benchmark designed to assess the capability of language models in comprehending ancient Chinese. ACLUE consists of 15 tasks cover a range of skills, spanning phonetic, lexical, syntactic, semantic, inference and knowledge. Through the evaluation of eight state-of-the-art LLMs, we observed a noticeable disparity in their performance between modern Chinese and ancient Chinese. Among the assessed models, ChatGLM2 demonstrates the most remarkable performance, achieving an average score of 37.4%. We have made our code and data public available.