# Pragmatic-Pedagogic Value Alignment

arXiv: https://arxiv.org/abs/1707.06354 · HTML: https://mlsanity.com/paper/1707.06354/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jul 2017.

## Built on by (most cited first, 30 of 36)

- [Human-in-the-Loop Imitation Learning using Remote Teleoperation](https://mlsanity.com/paper/2012.06733/) (2020)
- [Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences](https://mlsanity.com/paper/2002.09089/) (2020)
- [Building Machines that Learn and Think with People](https://mlsanity.com/paper/2408.03943/) (2024)
- [Pragmatic Instruction Following and Goal Assistance via Cooperative Language-Guided Inverse Planning](https://mlsanity.com/paper/2402.17930/) (2024)
- [Human irrationality: both bad and good for reward inference](https://mlsanity.com/paper/2111.06956/) (2021)
- [Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humans](https://mlsanity.com/paper/2209.13020/) (2022)
- [Cognitive science as a source of forward and inverse models of human decisions for robotics and control](https://mlsanity.com/paper/2109.00127/) (2021)
- [How to talk so AI will learn: Instructions, descriptions, and autonomy](https://mlsanity.com/paper/2206.07870/) (2022)
- [Inferring the Goals of Communicating Agents from Actions and Instructions](https://mlsanity.com/paper/2306.16207/) (2023)
- [An Efficient, Generalized Bellman Update For Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1806.03820/) (2018)
- [Metaethical Perspectives on 'Benchmarking' AI Ethics](https://mlsanity.com/paper/2204.05151/) (2022)
- [AssistanceZero: Scalably Solving Assistance Games](https://mlsanity.com/paper/2504.07091/) (2025)
- [Learning Altruistic Behaviours in Reinforcement Learning without External Rewards](https://mlsanity.com/paper/2107.09598/) (2021)
- [Emergence of Theory of Mind Collaboration in Multiagent Systems](https://mlsanity.com/paper/2110.00121/) (2021)
- [Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Input](https://mlsanity.com/paper/2405.14769/) (2024)
- [Iterative Teacher-Aware Learning](https://mlsanity.com/paper/2110.00137/) (2021)
- [Pragmatic Embodied Spoken Instruction Following in Human-Robot Collaboration with Theory of Mind](https://mlsanity.com/paper/2409.10849/) (2024)
- [Enhancing Human Experience in Human-Agent Collaboration: A Human-Centered Modeling Approach Based on Positive Human Gain](https://mlsanity.com/paper/2401.16444/) (2024)
- [Attention Schema in Neural Agents](https://mlsanity.com/paper/2305.17375/) (2023)
- [Communication-Efficient Desire Alignment for Embodied Agent-Human Adaptation](https://mlsanity.com/paper/2505.22503/) (2025)
- [Explaining Reward Functions to Humans for Better Human-Robot Collaboration](https://mlsanity.com/paper/2110.04192/) (2021)
- [Reinforcement Learning and Machine ethics:a systematic review](https://mlsanity.com/paper/2407.02425/) (2024)
- [Dynamically Switching Human Prediction Models for Efficient Planning](https://mlsanity.com/paper/2103.07815/) (2021)
- [Pragmatically Learning from Pedagogical Demonstrations in Multi-Goal Environments](https://mlsanity.com/paper/2206.04546/) (2022)
- [Goal Alignment: A Human-Aware Account of Value Alignment Problem](https://mlsanity.com/paper/2302.00813/) (2023)
- [Understanding Subjectivity through the Lens of Motivational Context in Model-Generated Image Satisfaction](https://mlsanity.com/paper/2403.05576/) (2024)
- [Explainable Moral Values: a neuro-symbolic approach to value classification](https://mlsanity.com/paper/2410.12631/) (2024)
- [Research Superalignment Should Advance Now with Alternating Competence and Conformity Optimization](https://mlsanity.com/paper/2503.07660/) (2025)
- [Beyond Algorethics: Addressing the Ethical and Anthropological Challenges of AI Recommender Systems](https://mlsanity.com/paper/2507.16430/) (2025)
- [A Roadmap for Robust End-to-End Alignment](https://mlsanity.com/paper/1809.01036/) (2018)

## Abstract (arXiv)

As intelligent systems gain autonomy and capability, it becomes vital to ensure that their objectives match those of their human users; this is known as the value-alignment problem. In robotics, value alignment is key to the design of collaborative robots that can integrate into human workflows, successfully inferring and adapting to their users' objectives as they go. We argue that a meaningful solution to value alignment must combine multi-agent decision theory with rich mathematical models of human cognition, enabling robots to tap into people's natural collaborative capabilities. We present a solution to the cooperative inverse reinforcement learning (CIRL) dynamic game based on well-established cognitive models of decision making and theory of mind. The solution captures a key reciprocity relation: the human will not plan her actions in isolation, but rather reason pedagogically about how the robot might learn from them; the robot, in turn, can anticipate this and interpret the human's actions pragmatically. To our knowledge, this work constitutes the first formal analysis of value alignment grounded in empirically validated cognitive models.