# Bits of Grass: Does GPT already know how to write like Whitman?

arXiv: https://arxiv.org/abs/2305.11064 · HTML: https://mlsanity.com/paper/2305.11064/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 12 references that are in the map.
- First posted on arXiv 10 May 2023.

## Builds on (most cited first, 3 of 3)

- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [Evaluating Verifiability in Generative Search Engines](https://mlsanity.com/paper/2304.09848/) (2023)

## Built on by (most cited first, 13 of 13)

- [A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing](https://mlsanity.com/paper/2310.08433/) (2023)
- [AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text](https://mlsanity.com/paper/2410.04265/) (2024)
- [LLM-based multi-agent poetry generation in non-cooperative environments](https://mlsanity.com/paper/2409.03659/) (2024)
- [Creative Beam Search: LLM-as-a-Judge For Improving Response Generation](https://mlsanity.com/paper/2405.00099/) (2024)
- [Subtle Biases Need Subtler Measures: Dual Metrics for Evaluating Representative and Affinity Bias in Large Language Models](https://mlsanity.com/paper/2405.14555/) (2024)
- [PhonologyBench: Evaluating Phonological Skills of Large Language Models](https://mlsanity.com/paper/2404.02456/) (2024)
- [Latent Planning Emerges with Scale](https://mlsanity.com/paper/2604.12493/) (2026)
- [Guided scenarios with simulated expert personae: a remarkable strategy to perform cognitive work](https://mlsanity.com/paper/2306.03104/) (2023)
- [Vietnamese Poem Generation & The Prospect Of Cross-Language Poem-To-Poem Translation](https://mlsanity.com/paper/2401.01078/) (2024)
- [POEMetric: The Last Stanza of Humanity](https://mlsanity.com/paper/2604.03695/) (2026)
- [Thinking Outside the (Gray) Box: A Context-Based Score for Assessing Value and Originality in Neural Text Generation](https://mlsanity.com/paper/2502.13207/) (2025)
- [Can Large Language Models Outperform Non-Experts in Poetry Evaluation? A Comparative Study Using the Consensual Assessment Technique](https://mlsanity.com/paper/2502.19064/) (2025)
- [BIPro: Zero-shot Chinese Poem Generation via Block Inverse Prompting Constrained Generation Framework](https://mlsanity.com/paper/2411.13237/) (2024)

## Abstract (arXiv)

This study examines the ability of GPT-3.5, GPT-3.5-turbo (ChatGPT) and GPT-4 models to generate poems in the style of specific authors using zero-shot and many-shot prompts (which use the maximum context length of 8192 tokens). We assess the performance of models that are not fine-tuned for generating poetry in the style of specific authors, via automated evaluation. Our findings indicate that without fine-tuning, even when provided with the maximum number of 17 poem examples (8192 tokens) in the prompt, these models do not generate poetry in the desired style.