# Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasets

arXiv: https://arxiv.org/abs/2406.18906 · HTML: https://mlsanity.com/paper/2406.18906/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 59 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Builds on (most cited first, 10 of 10)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research](https://mlsanity.com/paper/2402.00159/) (2024)
- [What's In My Big Data?](https://mlsanity.com/paper/2310.20707/) (2023)
- [Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4](https://mlsanity.com/paper/2305.00118/) (2023)
- [PoeLM: A Meter- and Rhyme-Controllable Language Model for Unsupervised Poetry Generation](https://mlsanity.com/paper/2205.12206/) (2022)
- [Investigating Robustness of Dialog Models to Popular Figurative Language Constructs](https://mlsanity.com/paper/2110.00687/) (2021)
- [CharPoet: A Chinese Classical Poetry Generation System Based on Token-free LLM](https://mlsanity.com/paper/2401.03512/) (2024)
- [Evaluating Diversity in Automatic Poetry Generation](https://mlsanity.com/paper/2406.15267/) (2024)
- [Metrical Tagging in the Wild: Building and Annotating Poetry Corpora with Rhythmic Features](https://mlsanity.com/paper/2102.08858/) (2021)
- [ALBERTI, a Multilingual Domain Specific Language Model for Poetry Analysis](https://mlsanity.com/paper/2307.01387/) (2023)

## Built on by (most cited first, 18 of 18)

- [Survey of Cultural Awareness in Language Models: Text and Beyond](https://mlsanity.com/paper/2411.00860/) (2024)
- [Does ChatGPT Have a Poetic Style?](https://mlsanity.com/paper/2410.15299/) (2024)
- [Provocations from the Humanities for Generative AI Research](https://mlsanity.com/paper/2502.19190/) (2025)
- [On Classification with Large Language Models in Cultural Analytics](https://mlsanity.com/paper/2410.12029/) (2024)
- [Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Models](https://mlsanity.com/paper/2503.12072/) (2025)
- [Multimodal Conversation Structure Understanding](https://mlsanity.com/paper/2505.17536/) (2025)
- [so much depends / upon / a whitespace: Why Whitespace Matters for Poets and LLMs](https://mlsanity.com/paper/2510.16713/) (2025)
- [Societal Impacts Research Requires Benchmarks for Creative Composition Tasks](https://mlsanity.com/paper/2504.06549/) (2025)
- [Says Who? Effective Zero-Shot Annotation of Focalization](https://mlsanity.com/paper/2409.11390/) (2024)
- [POEMetric: The Last Stanza of Humanity](https://mlsanity.com/paper/2604.03695/) (2026)
- [Automated Evaluation of Meter and Rhyme in Russian Generative and Human-Authored Poetry](https://mlsanity.com/paper/2502.20931/) (2025)
- [MetricalARGS: A Taxonomy for Studying Metrical Poetry with LLMs](https://mlsanity.com/paper/2510.08188/) (2025)
- [GhazalBench: Canonical Verse Access in LLMs across Persian Ghazals and Shakespearean Sonnets](https://mlsanity.com/paper/2603.09979/) (2026)
- [Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025](https://mlsanity.com/paper/2606.02255/) (2026)
- [Translating Classical Poetry into Modern Prose](https://mlsanity.com/paper/2606.02806/) (2026)
- [Phun-Bench: Evaluating LLMs on Phonological Understanding in Chinese](https://mlsanity.com/paper/2606.07300/) (2026)
- [Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?](https://mlsanity.com/paper/2606.30556/) (2026)
- [Neo-Classic: A Benchmark for Evaluating Linguistic-Aesthetic Reasoning in Classical Chinese Poetry](https://mlsanity.com/paper/2609.19154/) (2026)

## Abstract (arXiv)

Large language models (LLMs) can now generate and recognize poetry. But what do LLMs really know about poetry? We develop a task to evaluate how well LLMs recognize one aspect of English-language poetry--poetic form--which captures many different poetic features, including rhyme scheme, meter, and word or line repetition. By using a benchmark dataset of over 4.1k human expert-annotated poems, we show that state-of-the-art LLMs can successfully identify both common and uncommon fixed poetic forms--such as sonnets, sestinas, and pantoums--with surprisingly high accuracy. However, performance varies significantly by poetic form; the models struggle to identify unfixed poetic forms, especially those based on topic or visual features. We additionally measure how many poems from our benchmark dataset are present in popular pretraining datasets or memorized by GPT-4, finding that pretraining presence and memorization may improve performance on this task, but results are inconclusive. We release a benchmark evaluation dataset with 1.4k public domain poems and form annotations, results of memorization experiments and data audits, and code.