mlsanityData to 14 Oct 2026

Training Value-Aligned Reinforcement Learning Agents Using a Normative Prior

arXiv 2104.09469 ↗First posted 19 Apr 2021cs.LG cs.AI cs.HCcited by 14

Not on the map. Why?

Abstract (arXiv)

As more machine learning agents interact with humans, it is increasingly a prospect that an agent trained to perform a task optimally, using only a measure of task performance as feedback, can violate societal norms for acceptable behavior or cause harm. Value alignment is a property of intelligent agents wherein they solely pursue non-harmful behaviors or human-beneficial goals. We introduce an approach to value-aligned reinforcement learning, in which we train an agent with two reward signals: a standard task performance reward, plus a normative behavior reward. The normative behavior reward is derived from a value-aligned prior model previously shown to classify text as normative or non-normative. We show how variations on a policy shaping technique can balance these two sources of reward and produce policies that are both effective and perceived as being more normative. We test our value-alignment technique on three interactive text-based worlds; each world is designed specifically to challenge agents with a task as well as provide opportunities to deviate from the task to engage in normative and/or altruistic behavior.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 32 of its 57 references that are in the map. First posted on arXiv 19 Apr 2021.

Builds on

32 of 57 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  7. 2016Generative Adversarial Imitation Learningcited by 1,597
  8. 2015Bidirectional LSTM-CRF Models for Sequence Taggingcited by 546
  9. 2018TextWorld: A Learning Environment for Text-based Gamescited by 310
  10. 2016Transfer Learning for Low-Resource Neural Machine Translationcited by 188
  11. 2017AI Safety Gridworldscited by 110
  12. 2017Third-Person Imitation Learningcited by 103
20 more
  1. 2020Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotescited by 101
  2. 2015Deep Reinforcement Learning with a Natural Language Action Spacecited by 88
  3. 2020Graph Constrained Reinforcement Learning for Natural Language Action Spacescited by 80
  4. 2018Learn What Not to Learn: Action Elimination with Deep Reinforcement Learningcited by 77
  5. 2016Research Priorities for Robust and Beneficial Artificial Intelligencecited by 74
  6. 2015Language Understanding for Text-based Games Using Deep Reinforcement Learningcited by 71
  7. 2016Model-Free Imitation Learning with Policy Optimizationcited by 52
  8. 2020How to Avoid Being Eaten by a Grue: Structured Exploration Strategies for Textual Worldscited by 38
  9. 2019Story Realization: Expanding Plot Events into Sentencescited by 35
  10. 2020Deep Reinforcement Learning with Stacked Hierarchical Attention for Text-based Gamescited by 22
  11. 2020Playing Text-Based Games with Common Sensecited by 21
  12. 2019Learning Norms from Stories: A Prior for Value Aligned Agentscited by 19
  13. 2019LeDeepChef: Deep Reinforcement Learning Agent for Families of Text-Based Gamescited by 17
  14. 2019Comprehensible Context-driven Text Game Playingcited by 10
  15. 2017Explore, Exploit or Listen: Combining Human Feedback and Policy Model to Speed up Deep Reinforcement Learning in 3D Worldscited by 8
  16. 2016Computational Narrative Intelligence: A Human-Centered Goal for Artificial Intelligencecited by 6
  17. 2018dAIrector: Automatic Story Beat Generation through Knowledge Synthesiscited by 4
  18. 2019Efficient Supervision for Robot Learning via Imitation, Simulation, and Adaptationcited by 4
  19. 2018Using reinforcement learning to learn how to play text-based gamescited by 3
  20. 2020Reducing Non-Normative Text Generation from Language Modelscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 4; 2024: 3; 2025: 2; 2026: 1.

14 papers.

Sort:

By area:

  1. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 182
  2. 2022Forecasting Future World Events with Neural Networkscited by 63
  3. 2022Aligning to Social Norms and Values in Interactive Narrativescited by 32
  4. 2024Interpretable Concept Bottlenecks to Align Reinforcement Learning Agentscited by 23
  5. 2024HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedbackcited by 15
  6. 2023Story Shaping: Teaching Agents Human-like Behavior with Storiescited by 7
  7. 2023Learning Human-like Representations to Enable Learning Human Valuescited by 7
  8. 2024Reinforcement Learning and Machine ethics:a systematic reviewcited by 3
  9. 2021Towards Interactive Reinforcement Learning with Intrinsic Feedbackcited by 2
  10. 2023Learning Not to Spoofcited by 2
  11. 2022An Analysis of Deep Reinforcement Learning Agents for Text-based Gamescited by 1
  12. 2025The Odyssey of the Fittest: Can Agents Survive and Still Be Good?cited by 1
2 more
  1. 2025HAVA: Hybrid Approach to Value-Alignment through Reward Weighing for Reinforcement Learningcited by 0
  2. 2026Expected Moral Shortfall for Ethical Competence in Decision-making Modelscited by 0