mlsanityData to 14 Oct 2026

Learning Norms from Stories: A Prior for Value Aligned Agents

arXiv 1912.03553 ↗First posted 7 Dec 2019cs.AI cs.CL cs.LGcited by 19

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Value alignment is a property of an intelligent agent indicating that it can only pursue goals and activities that are beneficial to humans. Traditional approaches to value alignment use imitation learning or preference learning to infer the values of humans by observing their behavior. We introduce a complementary technique in which a value aligned prior is learned from naturally occurring stories which encode societal norms. Training data is sourced from the childrens educational comic strip, Goofus and Gallant. In this work, we train multiple machine learning models to classify natural language descriptions of situations found in the comic strip as normative or non normative by identifying if they align with the main characters behavior. We also report the models performance when transferring to two unrelated tasks with little to no additional training on the new task.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 15 of its 40 references that are in the map. First posted on arXiv 7 Dec 2019.

Builds on

15 of 40 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  6. 2016Generative Adversarial Imitation Learningcited by 1,597
  7. 2015Bidirectional LSTM-CRF Models for Sequence Taggingcited by 546
  8. 2016Cooperative Inverse Reinforcement Learningcited by 300
  9. 2016Transfer Learning for Low-Resource Neural Machine Translationcited by 188
  10. 2017Third-Person Imitation Learningcited by 103
  11. 2016Research Priorities for Robust and Beneficial Artificial Intelligencecited by 74
  12. 2019Story Realization: Expanding Plot Events into Sentencescited by 35
3 more
  1. 2016Computational Narrative Intelligence: A Human-Centered Goal for Artificial Intelligencecited by 6
  2. 2018dAIrector: Automatic Story Beat Generation through Knowledge Synthesiscited by 4
  3. 2019Efficient Supervision for Robot Learning via Imitation, Simulation, and Adaptationcited by 4

Built on by

Cited by year

Citing papers per year: 2021: 4; 2022: 3; 2023: 8; 2024: 3; 2025: 1.

19 papers.

Sort:

By area:

  1. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 87
  2. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 54
  3. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Modelscited by 45
  4. 2021HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networkscited by 38
  5. 2022Aligning to Social Norms and Values in Interactive Narrativescited by 32
  6. 2021Learning Knowledge Graph-based World Models of Textual Environmentscited by 31
  7. 2022Multimodal Knowledge Alignment with Reinforcement Learningcited by 30
  8. 2021MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learningcited by 25
  9. 2021Training Value-Aligned Reinforcement Learning Agents Using a Normative Priorcited by 14
  10. 2022Robust Preference Learning for Storytelling via Contrastive Reinforcement Learningcited by 14
  11. 2024Value Imprint: A Technique for Auditing the Human Values Embedded in RLHF Datasetscited by 11
  12. 2023Decolonial AI Alignment: Openness, Vi'sedsa-Dharma, and Including Excluded Knowledgescited by 10
7 more
  1. 2023Story Shaping: Teaching Agents Human-like Behavior with Storiescited by 7
  2. 2023Author as Character and Narrator: Deconstructing Personal Narratives from the r/AmITheAsshole Reddit Communitycited by 6
  3. 2024Norm Violation Detection in Multi-Agent Systems using Large Language Models: A Pilot Studycited by 6
  4. 2023Fairness and Sequential Decision Making: Limits, Lessons, and Opportunitiescited by 5
  5. 2024Intuitions of Compromise: Utilitarianism vs. Contractualismcited by 1
  6. 2025Evaluating Human-LLM Representation Alignment: A Case Study on Affective Sentence Generation for Augmentative and Alternative Communicationcited by 1
  7. 2023Multi-Value Alignment in Normative Multi-Agent System: Evolutionary Optimisation Approachcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Contextual Moral Value Alignment Through Context-Based Aggregation
  2. 2025HAVA: Hybrid Approach to Value-Alignment through Reward Weighing for Reinforcement Learning
  3. 2026Enhancing Value Alignment of LLMs with Multi-agent system and Combinatorial Fusion
  4. 2026Toward a Theory of Value in AI Alignment
  5. 2024Moral Alignment for LLM Agents
  6. 2024Democratizing Reward Design for Personal and Representative Value-Alignment
  7. 2024What are human values, and how do we align AI to them?
  8. 2026Contextual Value Alignment via Multilayer Combinatorial Fusion