# Can ChatGPT Really Understand Modern Chinese Poetry?

arXiv: https://arxiv.org/abs/2603.20851 · HTML: https://mlsanity.com/paper/2603.20851/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 58 references that are in the map.
- First posted on arXiv 21 Mar 2026.

## Builds on (most cited first, 16 of 16)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023)
- [Why We Need New Evaluation Metrics for NLG](https://mlsanity.com/paper/1707.06875/) (2017)
- [ElectionSim: Massive Population Election Simulation Powered by Large Language Model Driven Agents](https://mlsanity.com/paper/2410.20746/) (2024)
- [How to Design Translation Prompts for ChatGPT: An Empirical Study](https://mlsanity.com/paper/2304.02182/) (2023)
- [PRE: A Peer Review Based Large Language Model Evaluator](https://mlsanity.com/paper/2401.15641/) (2024)
- [Don't Go Far Off: An Empirical Study on Neural Poetry Translation](https://mlsanity.com/paper/2109.02972/) (2021)
- [What is the Best Way for ChatGPT to Translate Poetry?](https://mlsanity.com/paper/2406.03450/) (2024)
- [Benchmarking the Detection of LLMs-Generated Modern Chinese Poetry](https://mlsanity.com/paper/2509.01620/) (2025)
- [Can AI Write Classical Chinese Poetry like Humans? An Empirical Study Inspired by Turing Test](https://mlsanity.com/paper/2401.04952/) (2024)
- [Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese](https://mlsanity.com/paper/2505.22645/) (2025)
- [Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing](https://mlsanity.com/paper/2510.04120/) (2025)
- [Preconditioned Test-Time Adaptation for Out-of-Distribution Debiasing in Narrative Generation](https://mlsanity.com/paper/2603.13683/) (2026)

## Built on by (most cited first, 4 of 4)

- [Who Wrote This Line? Evaluating the Detection of LLM-Generated Classical Chinese Poetry](https://mlsanity.com/paper/2604.10101/) (2026)
- [Seeing the Poem: Image-Semantic Detection of AI-Generated Modern Chinese Poetry with MLLMs](https://mlsanity.com/paper/2605.22654/) (2026)
- [Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?](https://mlsanity.com/paper/2606.30556/) (2026)
- [Do Large Language Models Perform Well on Comprehending Poetic Logic in Modern Chinese Poetry?](https://mlsanity.com/paper/2608.21827/) (2026)

## Abstract (arXiv)

ChatGPT has demonstrated remarkable capabilities on both poetry generation and translation, yet its ability to truly understand poetry remains unexplored. Previous poetry-related work merely analyzed experimental outcomes without addressing fundamental issues of comprehension. This paper introduces a comprehensive framework for evaluating ChatGPT's understanding of modern poetry. We collaborated with professional poets to evaluate ChatGPT's interpretation of modern Chinese poems by different poets along multiple dimensions. Evaluation results show that ChatGPT's interpretations align with the original poets' intents in over 73% of the cases. However, its understanding in certain dimensions, particularly in capturing poeticity, proved to be less satisfactory. These findings highlight the effectiveness and necessity of our proposed framework. This study not only evaluates ChatGPT's ability to understand modern poetry but also establishes a solid foundation for future research on LLMs and their application to poetry-related tasks.