# Learning Norms from Stories: A Prior for Value Aligned Agents

arXiv: https://arxiv.org/abs/1912.03553 · HTML: https://mlsanity.com/paper/1912.03553/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 15 of its 40 references that are in the map.
- First posted on arXiv 7 Dec 2019.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1604.02201/) (2016)
- [Third-Person Imitation Learning](https://mlsanity.com/paper/1703.01703/) (2017)
- [Research Priorities for Robust and Beneficial Artificial Intelligence](https://mlsanity.com/paper/1602.03506/) (2016)
- [Story Realization: Expanding Plot Events into Sentences](https://mlsanity.com/paper/1909.03480/) (2019)
- [Computational Narrative Intelligence: A Human-Centered Goal for Artificial Intelligence](https://mlsanity.com/paper/1602.06484/) (2016)
- [dAIrector: Automatic Story Beat Generation through Knowledge Synthesis](https://mlsanity.com/paper/1811.03423/) (2018)
- [Efficient Supervision for Robot Learning via Imitation, Simulation, and Adaptation](https://mlsanity.com/paper/1904.07346/) (2019)

## Built on by (most cited first, 19 of 19)

- [Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback](https://mlsanity.com/paper/2303.05453/) (2023)
- [The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values](https://mlsanity.com/paper/2310.07629/) (2023)
- [From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Models](https://mlsanity.com/paper/2308.12014/) (2023)
- [HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networks](https://mlsanity.com/paper/2102.02515/) (2021)
- [Aligning to Social Norms and Values in Interactive Narratives](https://mlsanity.com/paper/2205.01975/) (2022)
- [Learning Knowledge Graph-based World Models of Textual Environments](https://mlsanity.com/paper/2106.09608/) (2021)
- [Multimodal Knowledge Alignment with Reinforcement Learning](https://mlsanity.com/paper/2205.12630/) (2022)
- [MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learning](https://mlsanity.com/paper/2201.00012/) (2021)
- [Training Value-Aligned Reinforcement Learning Agents Using a Normative Prior](https://mlsanity.com/paper/2104.09469/) (2021)
- [Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning](https://mlsanity.com/paper/2210.07792/) (2022)
- [Value Imprint: A Technique for Auditing the Human Values Embedded in RLHF Datasets](https://mlsanity.com/paper/2411.11937/) (2024)
- [Decolonial AI Alignment: Openness, Vi'sedsa-Dharma, and Including Excluded Knowledges](https://mlsanity.com/paper/2309.05030/) (2023)
- [Story Shaping: Teaching Agents Human-like Behavior with Stories](https://mlsanity.com/paper/2301.10107/) (2023)
- [Author as Character and Narrator: Deconstructing Personal Narratives from the r/AmITheAsshole Reddit Community](https://mlsanity.com/paper/2301.08104/) (2023)
- [Norm Violation Detection in Multi-Agent Systems using Large Language Models: A Pilot Study](https://mlsanity.com/paper/2403.16517/) (2024)
- [Fairness and Sequential Decision Making: Limits, Lessons, and Opportunities](https://mlsanity.com/paper/2301.05753/) (2023)
- [Intuitions of Compromise: Utilitarianism vs. Contractualism](https://mlsanity.com/paper/2410.05496/) (2024)
- [Evaluating Human-LLM Representation Alignment: A Case Study on Affective Sentence Generation for Augmentative and Alternative Communication](https://mlsanity.com/paper/2503.11881/) (2025)
- [Multi-Value Alignment in Normative Multi-Agent System: Evolutionary Optimisation Approach](https://mlsanity.com/paper/2305.07366/) (2023)

## Abstract (arXiv)

Value alignment is a property of an intelligent agent indicating that it can only pursue goals and activities that are beneficial to humans. Traditional approaches to value alignment use imitation learning or preference learning to infer the values of humans by observing their behavior. We introduce a complementary technique in which a value aligned prior is learned from naturally occurring stories which encode societal norms. Training data is sourced from the childrens educational comic strip, Goofus and Gallant. In this work, we train multiple machine learning models to classify natural language descriptions of situations found in the comic strip as normative or non normative by identifying if they align with the main characters behavior. We also report the models performance when transferring to two unrelated tasks with little to no additional training on the new task.