# Spoiler Alert: Narrative Forecasting as a Metric for Tension in LLM Storytelling

arXiv: https://arxiv.org/abs/2604.09854 · HTML: https://mlsanity.com/paper/2604.09854/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 45 references that are in the map.
- First posted on arXiv 10 Apr 2026.

## Builds on (most cited first, 27 of 27)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [Does Writing with Language Models Reduce Content Diversity?](https://mlsanity.com/paper/2309.05196/) (2023)
- [Art or Artifice? Large Language Models and the False Promise of Creativity](https://mlsanity.com/paper/2309.14556/) (2023)
- [Ghostbuster: Detecting Text Ghostwritten by Large Language Models](https://mlsanity.com/paper/2305.15047/) (2023)
- [EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models](https://mlsanity.com/paper/2312.06281/) (2023)
- [Are Large Language Models Capable of Generating Human-Level Narratives?](https://mlsanity.com/paper/2407.13248/) (2024)
- [Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation](https://mlsanity.com/paper/2208.11646/) (2022)
- [UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation](https://mlsanity.com/paper/2009.07602/) (2020)
- [Agents' Room: Narrative Generation through Multi-step Collaboration](https://mlsanity.com/paper/2410.02603/) (2024)
- [Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Edits](https://mlsanity.com/paper/2409.14509/) (2024)
- [Echoes in AI: Quantifying lack of plot diversity in LLM outputs](https://mlsanity.com/paper/2501.00273/) (2024)
- [SCORE: Story Coherence and Retrieval Enhancement for AI Narratives](https://mlsanity.com/paper/2503.23512/) (2025)
- [CharacterBox: Evaluating the Role-Playing Capabilities of LLMs in Text-Based Virtual Worlds](https://mlsanity.com/paper/2412.05631/) (2024)
- [Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking](https://mlsanity.com/paper/2409.15268/) (2024)
- [Learning to Reason for Long-Form Story Generation](https://mlsanity.com/paper/2503.22828/) (2025)
- [Pron vs Prompt: Can Large Language Models already Challenge a World-Class Fiction Author at Creative Text Writing?](https://mlsanity.com/paper/2407.01119/) (2024)
- [Creating Suspenseful Stories: Iterative Planning with Large Language Models](https://mlsanity.com/paper/2402.17119/) (2024)
- [Evaluating Creative Short Story Generation in Humans and Large Language Models](https://mlsanity.com/paper/2411.02316/) (2024)
- [Modelling Suspense in Short Stories as Uncertainty Reduction over Neural Representation](https://mlsanity.com/paper/2004.14905/) (2020)
- [Frankentext: Stitching random text fragments into long-form narratives](https://mlsanity.com/paper/2505.18128/) (2025)
- [The Reader is the Metric: How Textual Features and Reader Profiles Explain Conflicting Evaluations of AI Creative Writing](https://mlsanity.com/paper/2506.03310/) (2025)
- [KRISTEVA: Close Reading as a Novel Task for Benchmarking Interpretive Reasoning](https://mlsanity.com/paper/2505.09825/) (2025)
- [LLMs Exhibit Significantly Lower Uncertainty in Creative Writing Than Professional Writers](https://mlsanity.com/paper/2602.16162/) (2026)
- [Do Language Models Agree with Human Perceptions of Suspense in Stories?](https://mlsanity.com/paper/2508.15794/) (2025)

## Built on by (most cited first, 1 of 1)

- [Narrative Flattening: How Post-Training Compresses Thematic, Affective, and Stylistic Variation in LLM Fiction](https://mlsanity.com/paper/2605.27878/) (2026)

## Abstract (arXiv)

LLMs have so far failed both to generate consistently compelling stories and to recognize this failure--on the leading creative-writing benchmark (EQ-Bench), LLM judges rank zero-shot AI stories above New Yorker short stories, a gold standard for literary fiction. We argue that existing rubrics overlook a key dimension of compelling human stories: narrative tension. We introduce the 100-Endings metric, which walks through a story sentence by sentence: at each position, a model predicts how the story will end 100 times given only the text so far, and we measure tension as how often predictions fail to match the ground truth. Beyond the mismatch rate, the sentence-level curve also yields complementary statistics that track plot-level twists and revelations, such as the inflection rate, a geometric measure of how frequently the curve reverses direction. Unlike rubric-based judges, 100-Endings correctly ranks New Yorker stories far above LLM outputs. Grounded in narratological principles, we design a story-generation pipeline using structural constraints, including analysis of story templates, idea formulation, and narrative scaffolding. Our pipeline significantly increases narrative tension as measured by the 100-Endings metric, while maintaining performance on the EQ-Bench leaderboard.