# Neo-Classic: A Benchmark for Evaluating Linguistic-Aesthetic Reasoning in Classical Chinese Poetry

arXiv: https://arxiv.org/abs/2609.19154 · HTML: https://mlsanity.com/paper/2609.19154/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 22 Jul 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 16 of 16)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [Deep-speare: A Joint Neural Model of Poetic Language, Meter and Rhyme](https://mlsanity.com/paper/1807.03491/) (2018)
- [CharPoet: A Chinese Classical Poetry Generation System Based on Token-free LLM](https://mlsanity.com/paper/2401.03512/) (2024)
- [Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasets](https://mlsanity.com/paper/2406.18906/) (2024)
- [CCPM: A Chinese Classical Poetry Matching Dataset](https://mlsanity.com/paper/2106.01979/) (2021)
- [Generating Major Types of Chinese Classical Poetry in a Uniformed Framework](https://mlsanity.com/paper/2003.11528/) (2020)
- [Can Large Language Model Comprehend Ancient Chinese? A Preliminary Test on ACLUE](https://mlsanity.com/paper/2310.09550/) (2023)
- [F`ux`i: A Benchmark for Evaluating Language Models on Ancient Chinese Text Understanding and Generation](https://mlsanity.com/paper/2503.15837/) (2025)
- [PoeTone: A Framework for Constrained Generation of Structured Chinese Songci with LLMs](https://mlsanity.com/paper/2508.02515/) (2025)
- [Automated Evaluation of Meter and Rhyme in Russian Generative and Human-Authored Poetry](https://mlsanity.com/paper/2502.20931/) (2025)
- [MetricalARGS: A Taxonomy for Studying Metrical Poetry with LLMs](https://mlsanity.com/paper/2510.08188/) (2025)
- [Capabilities and Evaluation Biases of Large Language Models in Classical Chinese Poetry Generation: A Case Study on Tang Poetry](https://mlsanity.com/paper/2510.15313/) (2025)

## Abstract (arXiv)

While Large Language Models (LLMs) achieve high accuracy on established Classical Chinese Poetry benchmarks, it remains challenging to distinguish transferable Linguistic-Aesthetic Reasoning from reliance on familiar pre-training patterns. To address this issue, we introduce Neo-Classic, an evaluation benchmark that combines a constructionist Out-of-Sample (OOS) dataset with a suite of reverse understanding probes. Unlike traditional benchmarks that rely on verification or generation over historical corpora, Neo-Classic comprises strictly metrical poetry authored by contemporary experts, reducing the possibility of direct retrieval. We evaluate state-of-the-art models, including Qwen3-Max, Gemini-3-Pro, and DeepSeek-V3.2, across five behavioral probes designed to test hierarchical constraint satisfaction. Our results reveal two primary limitations. First, a performance gap of 20 to 50 percent emerges when models transition from historical to contemporary texts. Second, models exhibit substantial difficulties in discourse-level ordering tasks, with standard accuracy remaining low (0 to 13 percent). Although expert-level guidance improves the performance of reasoning-enhanced models to 36 percent, a notable gap with human experts persists. These findings suggest that while current LLMs capture local formal patterns, they struggle with global hierarchical planning required for robust Linguistic-Aesthetic Reasoning.