# Benchmarking the Detection of LLMs-Generated Modern Chinese Poetry

arXiv: https://arxiv.org/abs/2509.01620 · HTML: https://mlsanity.com/paper/2509.01620/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 65 references that are in the map.
- First posted on arXiv 1 Sep 2025.

## Builds on (most cited first, 19 of 19)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Release Strategies and the Social Impacts of Language Models](https://mlsanity.com/paper/1908.09203/) (2019)
- [GLTR: Statistical Detection and Visualization of Generated Text](https://mlsanity.com/paper/1906.04043/) (2019)
- [Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text](https://mlsanity.com/paper/2401.12070/) (2024)
- [Self-Consuming Generative Models Go MAD](https://mlsanity.com/paper/2307.01850/) (2023)
- [DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text](https://mlsanity.com/paper/2306.05540/) (2023)
- [Human heuristics for AI-generated language are flawed](https://mlsanity.com/paper/2206.07271/) (2022)
- [Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluation](https://mlsanity.com/paper/2304.01746/) (2023)
- [How to Design Translation Prompts for ChatGPT: An Empirical Study](https://mlsanity.com/paper/2304.02182/) (2023)
- [Who Wrote This? The Key to Zero-Shot LLM-Generated Text Detection Is GECScore](https://mlsanity.com/paper/2405.04286/) (2024)
- [RepreGuard: Detecting LLM-Generated Text by Revealing Hidden Representation Patterns](https://mlsanity.com/paper/2508.13152/) (2025)
- [Does Human Collaboration Enhance the Accuracy of Identifying LLM-Generated Deepfake Texts?](https://mlsanity.com/paper/2304.01002/) (2023)
- [Introducing Aspects of Creativity in Automatic Poetry Generation](https://mlsanity.com/paper/2002.02511/) (2020)
- [What is the Best Way for ChatGPT to Translate Poetry?](https://mlsanity.com/paper/2406.03450/) (2024)
- [Leveraging Explainable AI for LLM Text Attribution: Differentiating Human-Written and Multiple LLMs-Generated Text](https://mlsanity.com/paper/2501.03212/) (2025)
- [The Imitation Game: Detecting Human and AI-Generated Texts in the Era of ChatGPT and BARD](https://mlsanity.com/paper/2307.12166/) (2023)
- [Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese](https://mlsanity.com/paper/2505.22645/) (2025)
- [LLMCL-GEC: Advancing Grammatical Error Correction with LLM-Driven Curriculum Learning](https://mlsanity.com/paper/2412.12541/) (2024)
- [FOCUS: Forging Originality through Contrastive Use in Self-Plagiarism for Language Models](https://mlsanity.com/paper/2406.00839/) (2024)

## Built on by (most cited first, 10 of 10)

- [Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing](https://mlsanity.com/paper/2510.04120/) (2025)
- [Preconditioned Test-Time Adaptation for Out-of-Distribution Debiasing in Narrative Generation](https://mlsanity.com/paper/2603.13683/) (2026)
- [Can ChatGPT Really Understand Modern Chinese Poetry?](https://mlsanity.com/paper/2603.20851/) (2026)
- [Decoding AI Authorship: Can LLMs Truly Mimic Human Style Across Literature and Politics?](https://mlsanity.com/paper/2603.23219/) (2026)
- [Who Wrote This Line? Evaluating the Detection of LLM-Generated Classical Chinese Poetry](https://mlsanity.com/paper/2604.10101/) (2026)
- [Seeing the Poem: Image-Semantic Detection of AI-Generated Modern Chinese Poetry with MLLMs](https://mlsanity.com/paper/2605.22654/) (2026)
- [CreAgentive: An Agent Workflow Driven Multi-Category Creative Generation Engine](https://mlsanity.com/paper/2509.26461/) (2025)
- [Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?](https://mlsanity.com/paper/2606.30556/) (2026)
- [Do Large Language Models Perform Well on Comprehending Poetic Logic in Modern Chinese Poetry?](https://mlsanity.com/paper/2608.21827/) (2026)
- [Acmite: Mitigating Gender Bias in LLMs through Concept-Guided Mutual Information](https://mlsanity.com/paper/2610.01696/) (2026)

## Abstract (arXiv)

The rapid development of advanced large language models (LLMs) has made AI-generated text indistinguishable from human-written text. Previous work on detecting AI-generated text has made effective progress, but has not involved modern Chinese poetry. Due to the distinctive characteristics of modern Chinese poetry, it is difficult to identify whether a poem originated from humans or AI. The proliferation of AI-generated modern Chinese poetry has significantly disrupted the poetry ecosystem. Based on the urgency of identifying AI-generated poetry in the real Chinese world, this paper proposes a novel benchmark for detecting LLMs-generated modern Chinese poetry. We first construct a high-quality dataset, which includes both 800 poems written by six professional poets and 41,600 poems generated by four mainstream LLMs. Subsequently, we conduct systematic performance assessments of six detectors on this dataset. Experimental results demonstrate that current detectors cannot be used as reliable tools to detect modern Chinese poems generated by LLMs. The most difficult poetic features to detect are intrinsic qualities, especially style. The detection results verify the effectiveness and necessity of our proposed benchmark. Our work lays a foundation for future detection of AI-generated poetry.