# Goal-Directed Story Generation: Augmenting Generative Language Models with Reinforcement Learning

arXiv: https://arxiv.org/abs/2112.08593 · HTML: https://mlsanity.com/paper/2112.08593/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 56 references that are in the map.
- First posted on arXiv 16 Dec 2021.

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [Plan-And-Write: Towards Better Automatic Storytelling](https://mlsanity.com/paper/1811.05701/) (2018)
- [Strategies for Structuring Story Generation](https://mlsanity.com/paper/1902.01109/) (2019)
- [Enabling Language Models to Fill in the Blanks](https://mlsanity.com/paper/2005.05339/) (2020)
- [A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation](https://mlsanity.com/paper/2001.05139/) (2020)
- [PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Tracking](https://mlsanity.com/paper/2004.14967/) (2020)
- [Do Massively Pretrained Language Models Make Better Storytellers?](https://mlsanity.com/paper/1909.10705/) (2019)
- [Graph Constrained Reinforcement Learning for Natural Language Action Spaces](https://mlsanity.com/paper/2001.08837/) (2020)
- [Event Representations for Automated Story Generation with Deep Neural Nets](https://mlsanity.com/paper/1706.01331/) (2017)
- [Narrative Planning: Balancing Plot and Character](https://mlsanity.com/paper/1401.3841/) (2014)
- [Playing Text-Adventure Games with Graph-Based Deep Reinforcement Learning](https://mlsanity.com/paper/1812.01628/) (2018)
- [Controllable Neural Story Plot Generation via Reward Shaping](https://mlsanity.com/paper/1809.10736/) (2018)
- [How to Avoid Being Eaten by a Grue: Structured Exploration Strategies for Textual Worlds](https://mlsanity.com/paper/2006.07409/) (2020)
- [Automated Storytelling via Causal, Commonsense Plot Ordering](https://mlsanity.com/paper/2009.00829/) (2020)
- [Story Realization: Expanding Plot Events into Sentences](https://mlsanity.com/paper/1909.03480/) (2019)
- [Learning Knowledge Graph-based World Models of Textual Environments](https://mlsanity.com/paper/2106.09608/) (2021)
- [Inferring the Reader: Guiding Automated Story Generation with Commonsense Reasoning](https://mlsanity.com/paper/2105.01311/) (2021)
- [Narrative Interpolation for Generating and Understanding Stories](https://mlsanity.com/paper/2008.07466/) (2020)
- [DeepTingle](https://mlsanity.com/paper/1705.03557/) (2017)

## Built on by (most cited first, 14 of 14)

- [Quark: Controllable Text Generation with Reinforced Unlearning](https://mlsanity.com/paper/2205.13636/) (2022)
- [Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey](https://mlsanity.com/paper/2210.07700/) (2022)
- [Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning](https://mlsanity.com/paper/2210.07792/) (2022)
- [Transformers in Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.05979/) (2023)
- [LongStory: Coherent, Complete and Length Controlled Long story Generation](https://mlsanity.com/paper/2311.15208/) (2023)
- [Reinforcement Learning for Generative AI: A Survey](https://mlsanity.com/paper/2308.14328/) (2023)
- [Creative Wand: A System to Study Effects of Communications in Co-Creative Settings](https://mlsanity.com/paper/2208.02886/) (2022)
- [SWAG: Storytelling With Action Guidance](https://mlsanity.com/paper/2402.03483/) (2024)
- [CoRRPUS: Code-based Structured Prompting for Neurosymbolic Story Understanding](https://mlsanity.com/paper/2212.10754/) (2022)
- [Avoidance Decoding for Diverse Multi-Branch Story Generation](https://mlsanity.com/paper/2509.02170/) (2025)
- [Sequentially Controlled Text Generation](https://mlsanity.com/paper/2301.02299/) (2023)
- [Characterizing the Investigative Methods of Fictional Detectives with Large Language Models](https://mlsanity.com/paper/2505.07601/) (2025)
- [Shall We Play a Game? Language Models for Open-ended Wargames](https://mlsanity.com/paper/2509.17192/) (2025)
- [Multigenre AI-powered Story Composition](https://mlsanity.com/paper/2405.06685/) (2024)

## Abstract (arXiv)

The advent of large pre-trained generative language models has provided a common framework for AI story generation via sampling the model to create sequences that continue the story. However, sampling alone is insufficient for story generation. In particular, it is hard to direct a language model to create stories to reach a specific goal event. We present two automated techniques grounded in deep reinforcement learning and reward shaping to control the plot of computer-generated stories. The first utilizes proximal policy optimization to fine-tune an existing transformer-based language model to generate text continuations but also be goal-seeking. The second extracts a knowledge graph from the unfolding story, which is used by a policy network with graph attention to select a candidate continuation generated by a language model. We report on automated metrics pertaining to how often stories achieve a given goal event as well as human participant rankings of coherence and overall story quality compared to baselines and ablations.