# Story Shaping: Teaching Agents Human-like Behavior with Stories

arXiv: https://arxiv.org/abs/2301.10107 · HTML: https://mlsanity.com/paper/2301.10107/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 51 references that are in the map.
- First posted on arXiv 24 Jan 2023.

## Builds on (most cited first, 30 of 32)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [ScienceWorld: Is your Agent Smarter than a 5th Grader?](https://mlsanity.com/paper/2203.07540/) (2022)
- [TextWorld: A Learning Environment for Text-based Games](https://mlsanity.com/paper/1806.11532/) (2018)
- [Large-Scale Study of Curiosity-Driven Learning](https://mlsanity.com/paper/1808.04355/) (2018)
- [Visual Reinforcement Learning with Imagined Goals](https://mlsanity.com/paper/1807.04742/) (2018)
- [Skew-Fit: State-Covering Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/1903.03698/) (2019)
- [Interactive Fiction Games: A Colossal Adventure](https://mlsanity.com/paper/1909.05398/) (2019)
- [Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models](https://mlsanity.com/paper/1507.00814/) (2015)
- [Simple BERT Models for Relation Extraction and Semantic Role Labeling](https://mlsanity.com/paper/1904.05255/) (2019)
- [Learning to Speak and Act in a Fantasy Text Adventure Game](https://mlsanity.com/paper/1903.03094/) (2019)
- [Content Planning for Neural Story Generation with Aristotelian Rescoring](https://mlsanity.com/paper/2009.09870/) (2020)
- [Graph Constrained Reinforcement Learning for Natural Language Action Spaces](https://mlsanity.com/paper/2001.08837/) (2020)
- [Playing Text-Adventure Games with Graph-Based Deep Reinforcement Learning](https://mlsanity.com/paper/1812.01628/) (2018)
- [How to Avoid Being Eaten by a Grue: Structured Exploration Strategies for Textual Worlds](https://mlsanity.com/paper/2006.07409/) (2020)
- [How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds](https://mlsanity.com/paper/2010.00685/) (2020)
- [Aligning to Social Norms and Values in Interactive Narratives](https://mlsanity.com/paper/2205.01975/) (2022)
- [Inferring the Reader: Guiding Automated Story Generation with Commonsense Reasoning](https://mlsanity.com/paper/2105.01311/) (2021)
- [Bringing Stories Alive: Generating Interactive Fiction Worlds](https://mlsanity.com/paper/2001.10161/) (2020)
- [Hierarchical Reinforcement Learning with Hindsight](https://mlsanity.com/paper/1805.08180/) (2018)
- [Deep Reinforcement Learning with Stacked Hierarchical Attention for Text-based Games](https://mlsanity.com/paper/2010.11655/) (2020)
- [Playing Text-Based Games with Common Sense](https://mlsanity.com/paper/2012.02757/) (2020)
- [Learning Norms from Stories: A Prior for Value Aligned Agents](https://mlsanity.com/paper/1912.03553/) (2019)
- [Plug-and-Blend: A Framework for Controllable Story Generation with Blended Control Codes](https://mlsanity.com/paper/2104.04039/) (2021)
- [Inherently Explainable Reinforcement Learning in Natural Language](https://mlsanity.com/paper/2112.08907/) (2021)

## Built on by (most cited first, 7 of 7)

- [TALES: Text Adventure Learning Environment Suite](https://mlsanity.com/paper/2504.14128/) (2025)
- [Dialogue Shaping: Empowering Agents through NPC Interaction](https://mlsanity.com/paper/2307.15833/) (2023)
- [Thespian: Multi-Character Text Role-Playing Game Agents](https://mlsanity.com/paper/2308.01872/) (2023)
- [Ambient Adventures: Teaching ChatGPT on Developing Complex Stories](https://mlsanity.com/paper/2308.01734/) (2023)
- [A Mixture-of-Experts Approach to Few-Shot Task Transfer in Open-Ended Text Worlds](https://mlsanity.com/paper/2405.06059/) (2024)
- [Integrating Machine Learning into Belief-Desire-Intention Agents: Current Advances and Open Challenges](https://mlsanity.com/paper/2510.20641/) (2025)
- [ByteSized32Refactored: Towards an Extensible Interactive Text Games Corpus for LLM World Modeling and Evaluation](https://mlsanity.com/paper/2509.23979/) (2025)

## Abstract (arXiv)

Reward design for reinforcement learning agents can be difficult in situations where one not only wants the agent to achieve some effect in the world but where one also cares about how that effect is achieved. For example, we might wish for an agent to adhere to a tacit understanding of commonsense, align itself to a preference for how to behave for purposes of safety, or taking on a particular role in an interactive game. Storytelling is a mode for communicating tacit procedural knowledge. We introduce a technique, Story Shaping, in which a reinforcement learning agent infers tacit knowledge from an exemplar story of how to accomplish a task and intrinsically rewards itself for performing actions that make its current environment adhere to that of the inferred story world. Specifically, Story Shaping infers a knowledge graph representation of the world state from observations, and also infers a knowledge graph from the exemplar story. An intrinsic reward is generated based on the similarity between the agent's inferred world state graph and the inferred story world graph. We conducted experiments in text-based games requiring commonsense reasoning and shaping the behaviors of agents as virtual game characters.