mlsanityData to 14 Oct 2026

Can ChatGPT Really Understand Modern Chinese Poetry?

arXiv 2603.20851 ↗First posted 21 Mar 2026cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

ChatGPT has demonstrated remarkable capabilities on both poetry generation and translation, yet its ability to truly understand poetry remains unexplored. Previous poetry-related work merely analyzed experimental outcomes without addressing fundamental issues of comprehension. This paper introduces a comprehensive framework for evaluating ChatGPT's understanding of modern poetry. We collaborated with professional poets to evaluate ChatGPT's interpretation of modern Chinese poems by different poets along multiple dimensions. Evaluation results show that ChatGPT's interpretations align with the original poets' intents in over 73% of the cases. However, its understanding in certain dimensions, particularly in capturing poeticity, proved to be less satisfactory. These findings highlight the effectiveness and necessity of our proposed framework. This study not only evaluates ChatGPT's ability to understand modern poetry but also establishes a solid foundation for future research on LLMs and their application to poetry-related tasks.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 16 of its 58 references that are in the map. First posted on arXiv 21 Mar 2026.

Builds on

16 of 58 references.

References · 16

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  4. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  5. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  6. 2017Why We Need New Evaluation Metrics for NLGcited by 233
  7. 2024ElectionSim: Massive Population Election Simulation Powered by Large Language Model Driven Agentscited by 35
  8. 2023How to Design Translation Prompts for ChatGPT: An Empirical Studycited by 31
  9. 2024PRE: A Peer Review Based Large Language Model Evaluatorcited by 28
  10. 2021Don't Go Far Off: An Empirical Study on Neural Poetry Translationcited by 15
  11. 2024What is the Best Way for ChatGPT to Translate Poetry?cited by 14
  12. 2025Benchmarking the Detection of LLMs-Generated Modern Chinese Poetrycited by 10
4 more
  1. 2024Can AI Write Classical Chinese Poetry like Humans? An Empirical Study Inspired by Turing Testcited by 7
  2. 2025Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinesecited by 7
  3. 2025Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processingcited by 5
  4. 2026Preconditioned Test-Time Adaptation for Out-of-Distribution Debiasing in Narrative Generationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Who Wrote This Line? Evaluating the Detection of LLM-Generated Classical Chinese Poetrycited by 1
  2. 2026Seeing the Poem: Image-Semantic Detection of AI-Generated Modern Chinese Poetry with MLLMscited by 1
  3. 2026Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?cited by 0
  4. 2026Do Large Language Models Perform Well on Comprehending Poetic Logic in Modern Chinese Poetry?cited by 0