# Controllable Neural Story Plot Generation via Reward Shaping

arXiv: https://arxiv.org/abs/1809.10736 · HTML: https://mlsanity.com/paper/1809.10736/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Sep 2018.

## Built on by (most cited first, 30 of 49)

- [Quark: Controllable Text Generation with Reinforced Unlearning](https://mlsanity.com/paper/2205.13636/) (2022)
- [Offline RL for Natural Language Generation with Implicit Language Q Learning](https://mlsanity.com/paper/2206.11871/) (2022)
- [RL with KL penalties is better viewed as Bayesian inference](https://mlsanity.com/paper/2205.11275/) (2022)
- [A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models](https://mlsanity.com/paper/2201.05337/) (2022)
- [Aligning Language Models with Preferences through f-divergence Minimization](https://mlsanity.com/paper/2302.08215/) (2023)
- [LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models](https://mlsanity.com/paper/2311.18232/) (2023)
- [LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts](https://mlsanity.com/paper/2501.00274/) (2024)
- [On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting](https://mlsanity.com/paper/2206.00761/) (2022)
- [Beyond Text Generation: Supporting Writers with Continuous Automatic Text Summaries](https://mlsanity.com/paper/2208.09323/) (2022)
- [Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairs](https://mlsanity.com/paper/2101.10579/) (2021)
- [Inferring the Reader: Guiding Automated Story Generation with Commonsense Reasoning](https://mlsanity.com/paper/2105.01311/) (2021)
- [Automatic Story Generation: Challenges and Attempts](https://mlsanity.com/paper/2102.12634/) (2021)
- [Creativity and Machine Learning: A Survey](https://mlsanity.com/paper/2104.02726/) (2021)
- [Controllable Data Generation by Deep Learning: A Review](https://mlsanity.com/paper/2207.09542/) (2022)
- [Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approach](https://mlsanity.com/paper/2305.18427/) (2023)
- [GraphPlan: Story Generation by Planning with Event Graph](https://mlsanity.com/paper/2102.02977/) (2021)
- [Go Back in Time: Generating Flashbacks in Stories with Event Temporal Prompts](https://mlsanity.com/paper/2205.01898/) (2022)
- [Goal-Directed Story Generation: Augmenting Generative Language Models with Reinforcement Learning](https://mlsanity.com/paper/2112.08593/) (2021)
- [Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning](https://mlsanity.com/paper/2210.07792/) (2022)
- [GROVE: A Retrieval-augmented Complex Story Generation Framework with A Forest of Evidence](https://mlsanity.com/paper/2310.05388/) (2023)
- [Telling Stories through Multi-User Dialogue by Modeling Character Relations](https://mlsanity.com/paper/2105.15054/) (2021)
- [LongStory: Coherent, Complete and Length Controlled Long story Generation](https://mlsanity.com/paper/2311.15208/) (2023)
- [Energy-Based Models for Code Generation under Compilability Constraints](https://mlsanity.com/paper/2106.04985/) (2021)
- [Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges](https://mlsanity.com/paper/2308.00031/) (2023)
- [SuperHF: Supervised Iterative Learning from Human Feedback](https://mlsanity.com/paper/2310.16763/) (2023)
- [TropeTwist: Trope-based Narrative Structure Generation](https://mlsanity.com/paper/2204.09672/) (2022)
- [SWAG: Storytelling With Action Guidance](https://mlsanity.com/paper/2402.03483/) (2024)
- [Changing the Mind of Transformers for Topically-Controllable Language Generation](https://mlsanity.com/paper/2103.15335/) (2021)
- [Fabula Entropy Indexing: Objective Measures of Story Coherence](https://mlsanity.com/paper/2104.07472/) (2021)
- [CoRRPUS: Code-based Structured Prompting for Neurosymbolic Story Understanding](https://mlsanity.com/paper/2212.10754/) (2022)

## Abstract (arXiv)

Language-modeling--based approaches to story plot generation attempt to construct a plot by sampling from a language model (LM) to predict the next character, word, or sentence to add to the story. LM techniques lack the ability to receive guidance from the user to achieve a specific goal, resulting in stories that don't have a clear sense of progression and lack coherence. We present a reward-shaping technique that analyzes a story corpus and produces intermediate rewards that are backpropagated into a pre-trained LM in order to guide the model towards a given goal. Automated evaluations show our technique can create a model that generates story plots which consistently achieve a specified goal. Human-subject studies show that the generated stories have more plausible event ordering than baseline plot generation techniques.