# Evaluating Creative Short Story Generation in Humans and Large Language Models

arXiv: https://arxiv.org/abs/2411.02316 · HTML: https://mlsanity.com/paper/2411.02316/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 83 references that are in the map.
- First posted on arXiv 4 Nov 2024.

## Builds on (most cited first, 30 of 32)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [Self-Preference Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2410.21819/) (2024)
- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://mlsanity.com/paper/2406.18403/) (2024)
- [Plan-And-Write: Towards Better Automatic Storytelling](https://mlsanity.com/paper/1811.05701/) (2018)
- [Re3: Generating Longer Stories With Recursive Reprompting and Revision](https://mlsanity.com/paper/2210.06774/) (2022)
- [Does Writing with Language Models Reduce Content Diversity?](https://mlsanity.com/paper/2309.05196/) (2023)
- [Art or Artifice? Large Language Models and the False Promise of Creativity](https://mlsanity.com/paper/2309.14556/) (2023)
- [Objective-Reinforced Generative Adversarial Networks (ORGAN) for Sequence Generation Models](https://mlsanity.com/paper/1705.10843/) (2017)
- [CAN: Creative Adversarial Networks, Generating "Art" by Learning About Styles and Deviating from Style Norms](https://mlsanity.com/paper/1706.07068/) (2017)
- [Content Planning for Neural Story Generation with Aristotelian Rescoring](https://mlsanity.com/paper/2009.09870/) (2020)
- [Are Large Language Models Capable of Generating Human-Level Narratives?](https://mlsanity.com/paper/2407.13248/) (2024)
- [Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation](https://mlsanity.com/paper/2208.11646/) (2022)
- [Putting GPT-3's Creativity to the (Alternative Uses) Test](https://mlsanity.com/paper/2206.08932/) (2022)
- [MacGyver: Are Large Language Models Creative Problem Solvers?](https://mlsanity.com/paper/2311.09682/) (2023)
- [Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluation](https://mlsanity.com/paper/2405.13769/) (2024)
- [Visual Writing Prompts: Character-Grounded Story Generation with Curated Image Sequences](https://mlsanity.com/paper/2301.08571/) (2023)
- [Pron vs Prompt: Can Large Language Models already Challenge a World-Class Fiction Author at Creative Text Writing?](https://mlsanity.com/paper/2407.01119/) (2024)
- [Small Language Models can Outperform Humans in Short Creative Writing: A Study Comparing SLMs with Humans and LLMs](https://mlsanity.com/paper/2409.11547/) (2024)
- [Enhancing Creativity in Large Language Models through Associative Thinking Strategies](https://mlsanity.com/paper/2405.06715/) (2024)
- [Active Divergence with Generative Deep Learning -- A Survey and Taxonomy](https://mlsanity.com/paper/2107.05599/) (2021)
- [Creative Problem Solving in Large Language and Vision Models -- What Would it Take?](https://mlsanity.com/paper/2405.01453/) (2024)

## Built on by (most cited first, 21 of 21)

- [Learning to Reason for Long-Form Story Generation](https://mlsanity.com/paper/2503.22828/) (2025)
- [Frankentext: Stitching random text fragments into long-form narratives](https://mlsanity.com/paper/2505.18128/) (2025)
- [Pencils to Pixels: A Systematic Study of Creative Drawings across Children, Adults and AI](https://mlsanity.com/paper/2502.05999/) (2025)
- [Has the Creativity of Large-Language Models peaked? An analysis of inter- and intra-LLM variability](https://mlsanity.com/paper/2504.12320/) (2025)
- [ALAS: A Stateful Multi-LLM Agent Framework for Disruption-Aware Planning](https://mlsanity.com/paper/2505.12501/) (2025)
- [LLMs Exhibit Significantly Lower Uncertainty in Creative Writing Than Professional Writers](https://mlsanity.com/paper/2602.16162/) (2026)
- [Evaluating LLM Story Generation through Large-scale Network Analysis of Social Structures](https://mlsanity.com/paper/2510.18932/) (2025)
- [Through the Judge's Eyes: Inferred Thinking Traces Improve Reliability of LLM Raters](https://mlsanity.com/paper/2510.25860/) (2025)
- [Inducing Sustained Creativity and Diversity in Large Language Models](https://mlsanity.com/paper/2603.19519/) (2026)
- [Are LLMs becoming similarly creative? Evidence from three years of models](https://mlsanity.com/paper/2608.19437/) (2026)
- [Aether Weaver: Multimodal Affective Narrative Co-Generation with Dynamic Scene Graphs](https://mlsanity.com/paper/2507.21893/) (2025)
- [Spoiler Alert: Narrative Forecasting as a Metric for Tension in LLM Storytelling](https://mlsanity.com/paper/2604.09854/) (2026)
- [On the Creativity of AI Agents](https://mlsanity.com/paper/2604.13242/) (2026)
- [Ex Ante Evaluation of AI-Induced Idea Diversity Collapse](https://mlsanity.com/paper/2605.06540/) (2026)
- [NARRA-Gym for Evaluating Interactive Narrative Agents](https://mlsanity.com/paper/2605.08503/) (2026)
- [The Limits of Automatic Evaluation of Creativity in Large Language Models](https://mlsanity.com/paper/2608.23705/) (2026)
- [Galton's Law of Mediocrity: Why Large Language Models Regress to the Mean and Fail at Creativity in Advertising](https://mlsanity.com/paper/2509.25767/) (2025)
- [Evaluating Cultural Knowledge Processing in Large Language Models: A Cognitive Benchmarking Framework Integrating Retrieval-Augmented Generation](https://mlsanity.com/paper/2511.01649/) (2025)
- [Large Language Models Align with the Human Brain during Creative Thinking](https://mlsanity.com/paper/2604.03480/) (2026)
- [LLMs Should Incorporate Explicit Mechanisms for Human Empathy](https://mlsanity.com/paper/2604.10557/) (2026)
- [Where Models Converge and Humans Diverge: A Coverage Framework for Distributional Pluralism in Open-Ended Generation](https://mlsanity.com/paper/2608.05576/) (2026)

## Abstract (arXiv)

Story-writing is a fundamental aspect of human imagination, relying heavily on creativity to produce narratives that are novel, effective, and surprising. While large language models (LLMs) have demonstrated the ability to generate high-quality stories, their creative story-writing capabilities remain under-explored. In this work, we conduct a systematic analysis of creativity in short story generation across 60 LLMs and 60 people using a five-sentence cue-word-based creative story-writing task. We use measures to automatically evaluate model- and human-generated stories across several dimensions of creativity, including novelty, surprise, diversity, and linguistic complexity. We also collect creativity ratings and Turing Test classifications from non-expert and expert human raters and LLMs. Automated metrics show that LLMs generate stylistically complex stories, but tend to fall short in terms of novelty, surprise and diversity when compared to average human writers. Expert ratings generally coincide with automated metrics. However, LLMs and non-experts rate LLM stories to be more creative than human-generated stories. We discuss why and how these differences in ratings occur, and their implications for both human and artificial creativity.