# POEMetric: The Last Stanza of Humanity

arXiv: https://arxiv.org/abs/2604.03695 · HTML: https://mlsanity.com/paper/2604.03695/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 57 references that are in the map.
- First posted on arXiv 4 Apr 2026.

## Builds on (most cited first, 30 of 31)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://mlsanity.com/paper/2406.18403/) (2024)
- [ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Models](https://mlsanity.com/paper/2212.10474/) (2022)
- [PoeLM: A Meter- and Rhyme-Controllable Language Model for Unsupervised Poetry Generation](https://mlsanity.com/paper/2205.12206/) (2022)
- [CharPoet: A Chinese Classical Poetry Generation System Based on Token-free LLM](https://mlsanity.com/paper/2401.03512/) (2024)
- [Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasets](https://mlsanity.com/paper/2406.18906/) (2024)
- [Evaluating Diversity in Automatic Poetry Generation](https://mlsanity.com/paper/2406.15267/) (2024)
- [Generating Chinese Classical Poems with RNN Encoder-Decoder](https://mlsanity.com/paper/1604.01537/) (2016)
- [Introducing Aspects of Creativity in Automatic Poetry Generation](https://mlsanity.com/paper/2002.02511/) (2020)
- [Bits of Grass: Does GPT already know how to write like Whitman?](https://mlsanity.com/paper/2305.11064/) (2023)
- [GPT-based Generation for Classical Chinese Poetry](https://mlsanity.com/paper/1907.00151/) (2019)
- [Does ChatGPT Have a Poetic Style?](https://mlsanity.com/paper/2410.15299/) (2024)
- [Chinese Song Iambics Generation with Neural Attention-based Model](https://mlsanity.com/paper/1604.06274/) (2016)
- [Flexible and Creative Chinese Poetry Generation Using Neural Memory](https://mlsanity.com/paper/1705.03773/) (2017)
- [Can AI Write Classical Chinese Poetry like Humans? An Empirical Study Inspired by Turing Test](https://mlsanity.com/paper/2401.04952/) (2024)
- [A Multi-Modal Chinese Poetry Generation Model](https://mlsanity.com/paper/1806.09792/) (2018)
- [GPoeT-2: A GPT-2 Based Poem Generator](https://mlsanity.com/paper/2205.08847/) (2022)
- [Chinese Poetry Generation with a Salient-Clue Mechanism](https://mlsanity.com/paper/1809.04313/) (2018)
- [Vietnamese Poem Generation & The Prospect Of Cross-Language Poem-To-Poem Translation](https://mlsanity.com/paper/2401.01078/) (2024)
- [GPT Czech Poet: Generation of Czech Poetic Strophes with Language Models](https://mlsanity.com/paper/2407.12790/) (2024)
- [TACOMORE: Exploring a replicable prompting protocol for LLM-assisted corpus analysis](https://mlsanity.com/paper/2412.10139/) (2024)

## Built on by (most cited first, 2 of 2)

- [Limits of Reliability and Scaling in Language Models](https://mlsanity.com/paper/2607.14112/) (2026)
- [Characterizing Human-Likeness in AI Generated Poetry: A Zero-shot Classification Study](https://mlsanity.com/paper/2607.26221/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) can compose poetry, but how far are they from human poets? In this paper, we introduce POEMetric, the first comprehensive framework for poetry evaluation, examining 1) basic instruction-following abilities in generating poems according to a certain form and theme, 2) advanced abilities of showing creativity, lexical diversity, and idiosyncrasy, evoking emotional resonance, and using imagery and literary devices, and 3) general appraisal of the overall poem quality and estimation of authorship. We curated a human poem dataset - 203 English poems of 7 fixed forms annotated with meter, rhyme patterns and themes - and experimented with 30 LLMs for poetry generation based on the same forms and themes of the human data, totaling 6,090 LLM poems. Based on POEMetric, we assessed the performance of both human poets and LLMs through rule-based evaluation and LLM-as-a-judge, whose results were validated by human experts. Results show that, though the top model achieved high form accuracy (4.26 out of 5.00, with Gemini-2.5-Pro as a judge; same below) and theme alignment (4.99), all models failed to reach the same level of advanced abilities as human poets, who achieved unparalleled creativity (4.02), idiosyncrasy (3.95), emotional resonance (4.06), and skillful use of imagery (4.49) and literary devices (4.67). Humans also defeated the best-performing LLM in overall poem quality (4.22 vs. 3.20). As such, poetry generation remains a formidable challenge for LLMs. Data and codes are released at https://github.com/Bingru-Li/POEMetric.