# AC-EVAL: Evaluating Ancient Chinese Language Understanding in Large Language Models

arXiv: https://arxiv.org/abs/2403.06574 · HTML: https://mlsanity.com/paper/2403.06574/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 11 Mar 2024.

## Builds on (most cited first, 16 of 16)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [CMMLU: Measuring massive multitask language understanding in Chinese](https://mlsanity.com/paper/2306.09212/) (2023)
- [SuperCLUE: A Comprehensive Chinese Large Language Model Benchmark](https://mlsanity.com/paper/2307.15020/) (2023)
- [CLEVA: Chinese Language Models EVAluation Platform](https://mlsanity.com/paper/2308.04813/) (2023)
- [WYWEB: A NLP Evaluation Benchmark For Classical Chinese](https://mlsanity.com/paper/2305.14150/) (2023)
- [Evaluating the Generation Capabilities of Large Chinese Language Models](https://mlsanity.com/paper/2308.04823/) (2023)
- [Towards Effective Ancient Chinese Translation: Dataset, Model, and Evaluation](https://mlsanity.com/paper/2308.00240/) (2023)
- [That Slepen Al the Nyght with Open Ye! Cross-era Sequence Segmentation with Switch-memory](https://mlsanity.com/paper/2209.02967/) (2022)

## Built on by (most cited first, 5 of 5)

- [Survey of Cultural Awareness in Language Models: Text and Beyond](https://mlsanity.com/paper/2411.00860/) (2024)
- [SAGE: From Direct Answering to Evidence-Grounded Inference for Chinese Ancient Document Understanding](https://mlsanity.com/paper/2608.24011/) (2026)
- [AI Historian: Helping historians organize and verify person-centred temporal clues from dispersed historical narratives](https://mlsanity.com/paper/2608.29133/) (2026)
- [CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systems](https://mlsanity.com/paper/2609.22494/) (2026)
- [Does the Linguistic Register of Thinking Matter? Wenyan as a Compact Reasoning Register for Chinese LLMs](https://mlsanity.com/paper/2610.19741/) (2026)

## Abstract (arXiv)

Given the importance of ancient Chinese in capturing the essence of rich historical and cultural heritage, the rapid advancements in Large Language Models (LLMs) necessitate benchmarks that can effectively evaluate their understanding of ancient contexts. To meet this need, we present AC-EVAL, an innovative benchmark designed to assess the advanced knowledge and reasoning capabilities of LLMs within the context of ancient Chinese. AC-EVAL is structured across three levels of difficulty reflecting different facets of language comprehension: general historical knowledge, short text understanding, and long text comprehension. The benchmark comprises 13 tasks, spanning historical facts, geography, social customs, art, philosophy, classical poetry and prose, providing a comprehensive assessment framework. Our extensive evaluation of top-performing LLMs, tailored for both English and Chinese, reveals a substantial potential for enhancing ancient text comprehension. By highlighting the strengths and weaknesses of LLMs, AC-EVAL aims to promote their development and application forward in the realms of ancient Chinese language education and scholarly research. The AC-EVAL data and evaluation code are available at https://github.com/yuting-wei/AC-EVAL.