# Can AI Write Classical Chinese Poetry like Humans? An Empirical Study Inspired by Turing Test

arXiv: https://arxiv.org/abs/2401.04952 · HTML: https://mlsanity.com/paper/2401.04952/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 10 Jan 2024.

## Builds on (most cited first, 10 of 10)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [PoeLM: A Meter- and Rhyme-Controllable Language Model for Unsupervised Poetry Generation](https://mlsanity.com/paper/2205.12206/) (2022)
- [Introducing Aspects of Creativity in Automatic Poetry Generation](https://mlsanity.com/paper/2002.02511/) (2020)
- [An Iterative Polishing Framework based on Quality Aware Masked Language Model for Chinese Poetry Generation](https://mlsanity.com/paper/1911.13182/) (2019)
- [Chinese Poetry Generation with a Working Memory Model](https://mlsanity.com/paper/1809.04306/) (2018)
- [Can Machine Generate Traditional Chinese Poetry? A Feigenbaum Test](https://mlsanity.com/paper/1606.05829/) (2016)

## Built on by (most cited first, 7 of 7)

- [AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text](https://mlsanity.com/paper/2410.04265/) (2024)
- [Graph-based Confidence Calibration for Large Language Models](https://mlsanity.com/paper/2411.02454/) (2024)
- [Can ChatGPT Really Understand Modern Chinese Poetry?](https://mlsanity.com/paper/2603.20851/) (2026)
- [POEMetric: The Last Stanza of Humanity](https://mlsanity.com/paper/2604.03695/) (2026)
- [BIPro: Zero-shot Chinese Poem Generation via Block Inverse Prompting Constrained Generation Framework](https://mlsanity.com/paper/2411.13237/) (2024)
- [Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?](https://mlsanity.com/paper/2606.30556/) (2026)
- [Do Large Language Models Perform Well on Comprehending Poetic Logic in Modern Chinese Poetry?](https://mlsanity.com/paper/2608.21827/) (2026)

## Abstract (arXiv)

Some argue that the essence of humanity, such as creativity and sentiment, can never be mimicked by machines. This paper casts doubt on this belief by studying a vital question: Can AI compose poetry as well as humans? To answer the question, we propose ProFTAP, a novel evaluation framework inspired by Turing test to assess AI's poetry writing capability. We apply it on current large language models (LLMs) and find that recent LLMs do indeed possess the ability to write classical Chinese poems nearly indistinguishable from those of humans. We also reveal that various open-source LLMs can outperform GPT-4 on this task.