# Goal Alignment: A Human-Aware Account of Value Alignment Problem

arXiv: https://arxiv.org/abs/2302.00813 · HTML: https://mlsanity.com/paper/2302.00813/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 53 references that are in the map.
- First posted on arXiv 2 Feb 2023.

## Builds on (most cited first, 9 of 9)

- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [Inverse Reward Design](https://mlsanity.com/paper/1711.02827/) (2017)
- [Reward-rational (implicit) choice: A unifying formalism for reward learning](https://mlsanity.com/paper/2002.04833/) (2020)
- [Pragmatic-Pedagogic Value Alignment](https://mlsanity.com/paper/1707.06354/) (2017)
- [Symbols as a Lingua Franca for Bridging Human-AI Chasm for Explainable and Advisable AI Systems](https://mlsanity.com/paper/2109.09904/) (2021)
- [Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Inscrutable Representations](https://mlsanity.com/paper/2002.01080/) (2020)
- [Plan Explicability and Predictability for Robot Task Planning](https://mlsanity.com/paper/1511.08158/) (2015)
- [Model Elicitation through Direct Questioning](https://mlsanity.com/paper/2011.12262/) (2020)

## Built on by (most cited first, 1 of 1)

- [Expectation Alignment: Handling Reward Misspecification in the Presence of Expectation Mismatch](https://mlsanity.com/paper/2404.08791/) (2024)

## Abstract (arXiv)

Value alignment problems arise in scenarios where the specified objectives of an AI agent don't match the true underlying objective of its users. The problem has been widely argued to be one of the central safety problems in AI. Unfortunately, most existing works in value alignment tend to focus on issues that are primarily related to the fact that reward functions are an unintuitive mechanism to specify objectives. However, the complexity of the objective specification mechanism is just one of many reasons why the user may have misspecified their objective. A foundational cause for misalignment that is being overlooked by these works is the inherent asymmetry in human expectations about the agent's behavior and the behavior generated by the agent for the specified objective. To address this lacuna, we propose a novel formulation for the value alignment problem, named goal alignment that focuses on a few central challenges related to value alignment. In doing so, we bridge the currently disparate research areas of value alignment and human-aware planning. Additionally, we propose a first-of-its-kind interactive algorithm that is capable of using information generated under incorrect beliefs about the agent, to determine the true underlying goal of the user.