# Can LLMs Generate Good Stories? Insights and Challenges from a Narrative Planning Perspective

arXiv: https://arxiv.org/abs/2506.10161 · HTML: https://mlsanity.com/paper/2506.10161/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 54 references that are in the map.
- First posted on arXiv 11 Jun 2025.

## Builds on (most cited first, 22 of 22)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Plan-And-Write: Towards Better Automatic Storytelling](https://mlsanity.com/paper/1811.05701/) (2018)
- [LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks](https://mlsanity.com/paper/2402.01817/) (2024)
- [Art or Artifice? Large Language Models and the False Promise of Creativity](https://mlsanity.com/paper/2309.14556/) (2023)
- [Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thought](https://mlsanity.com/paper/2501.04682/) (2025)
- [Large Language Models and Games: A Survey and Roadmap](https://mlsanity.com/paper/2402.18659/) (2024)
- [Are Large Language Models Capable of Generating Human-Level Narratives?](https://mlsanity.com/paper/2407.13248/) (2024)
- [Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation](https://mlsanity.com/paper/2208.11646/) (2022)
- [Event Representations for Automated Story Generation with Deep Neural Nets](https://mlsanity.com/paper/1706.01331/) (2017)
- [Narrative Planning: Balancing Plot and Character](https://mlsanity.com/paper/1401.3841/) (2014)
- [Clingo = ASP + Control: Preliminary Report](https://mlsanity.com/paper/1405.3694/) (2014)
- [Experimental Narratives: A Comparison of Human Crowdsourced Storytelling and AI Storytelling](https://mlsanity.com/paper/2310.12902/) (2023)
- [Language as Reality: A Co-Creative Storytelling Game Experience in 1001 Nights using Generative AI](https://mlsanity.com/paper/2308.12915/) (2023)
- [StoryVerse: Towards Co-authoring Dynamic Plot with LLM-based Character Simulation via Narrative Planning](https://mlsanity.com/paper/2405.13042/) (2024)
- [What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation](https://mlsanity.com/paper/2408.14622/) (2024)
- [LLMs Can Plan Only If We Tell Them](https://mlsanity.com/paper/2501.13545/) (2025)
- [GPT for Games: A Scoping Review (2020-2023)](https://mlsanity.com/paper/2404.17794/) (2024)
- [Guiding and Diversifying LLM-Based Story Generation via Answer Set Programming](https://mlsanity.com/paper/2406.00554/) (2024)
- [WhatELSE: Shaping Narrative Spaces at Configurable Level of Abstraction for AI-bridged Interactive Storytelling](https://mlsanity.com/paper/2502.18641/) (2025)
- [Narrative Planning: Compilations to Classical Planning](https://mlsanity.com/paper/1401.5856/) (2014)

## Built on by (most cited first, 2 of 2)

- [StoryAlign: Evaluating and Training Reward Models for Story Generation](https://mlsanity.com/paper/2605.04831/) (2026)
- [The Challenge and Reward of Fair Play in Narrative: A Computational Approach](https://mlsanity.com/paper/2507.13841/) (2025)

## Abstract (arXiv)

Story generation has been a prominent application of Large Language Models (LLMs). However, understanding LLMs' ability to produce high-quality stories remains limited due to challenges in automatic evaluation methods and the high cost and subjectivity of manual evaluation. Computational narratology offers valuable insights into what constitutes a good story, which has been applied in the symbolic narrative planning approach to story generation. This work aims to deepen the understanding of LLMs' story generation capabilities by using them to solve narrative planning problems. We present a benchmark for evaluating LLMs on narrative planning based on literature examples, focusing on causal soundness, character intentionality, and dramatic conflict. Our experiments show that GPT-4 tier LLMs can generate causally sound stories at small scales, but planning with character intentionality and dramatic conflict remains challenging, requiring LLMs trained with reinforcement learning for complex reasoning. The results offer insights on the scale of stories that LLMs can generate while maintaining quality from different aspects. Our findings also highlight interesting problem solving behaviors and shed lights on challenges and considerations for applying LLM narrative planning in game environments.