# Machine Theory of Mind and the Structure of Human Values

arXiv: https://arxiv.org/abs/2505.20342 · HTML: https://mlsanity.com/paper/2505.20342/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 44 references that are in the map.
- First posted on arXiv 24 May 2025.

## Builds on (most cited first, 6 of 6)

- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress](https://mlsanity.com/paper/1806.06877/) (2018)
- [Goal Misgeneralization in Deep Reinforcement Learning](https://mlsanity.com/paper/2105.14111/) (2021)
- [Online Bayesian Goal Inference for Boundedly-Rational Planning Agents](https://mlsanity.com/paper/2006.07532/) (2020)
- [Solving the Baby Intuitions Benchmark with a Hierarchically Bayesian Theory of Mind](https://mlsanity.com/paper/2208.02914/) (2022)
- [A Review on Machine Theory of Mind](https://mlsanity.com/paper/2303.11594/) (2023)

## Abstract (arXiv)

Value learning is a crucial aspect of safe and ethical AI. This is primarily pursued by methods inferring human values from behaviour. However, humans care about much more than we are able to demonstrate through our actions. Consequently, an AI must predict the rest of our seemingly complex values from a limited sample. I call this the value generalization problem. In this paper, I argue that human values have a generative rational structure and that this allows us to solve the value generalization problem. In particular, we can use Bayesian Theory of Mind models to infer human values not only from behaviour, but also from other values. This has been obscured by the widespread use of simple utility functions to represent human values. I conclude that developing generative value-to-value inference is a crucial component of achieving a scalable machine theory of mind.