# Learning Human Rewards by Inferring Their Latent Intelligence Levels in Multi-Agent Games: A Theory-of-Mind Approach with Application to Driving Data

arXiv: https://arxiv.org/abs/2103.04289 · HTML: https://mlsanity.com/paper/2103.04289/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving.
- Builds on 16 of its 37 references that are in the map.
- First posted on arXiv 7 Mar 2021.

## Placed in

- End-to-end autonomous driving / [Imitation-learning end-to-end planners](https://mlsanity.com/driving/imitation-learning-end-to-end-planners/)
- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 16 of 16)

- [INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps](https://mlsanity.com/paper/1910.03088/) (2019)
- [A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models](https://mlsanity.com/paper/1611.03852/) (2016)
- [Batch Active Preference-Based Learning of Reward Functions](https://mlsanity.com/paper/1810.04303/) (2018)
- [Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1901.09207/) (2019)
- [Multi-Agent Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1907.13220/) (2019)
- [Hierarchical Game-Theoretic Planning for Autonomous Vehicles](https://mlsanity.com/paper/1810.05766/) (2018)
- [Efficient Sampling-Based Maximum Entropy Inverse Reinforcement Learning with Application to Autonomous Driving](https://mlsanity.com/paper/2006.13704/) (2020)
- [Probabilistic Prediction of Interactive Driving Behavior via Hierarchical Inverse Reinforcement Learning](https://mlsanity.com/paper/1809.02926/) (2018)
- [Game-Theoretic Modeling of Driver and Vehicle Interactions for Verification and Validation of Autonomous Vehicle Control Systems](https://mlsanity.com/paper/1608.08589/) (2016)
- [On the Utility of Model Learning in HRI](https://mlsanity.com/paper/1901.01291/) (2019)
- [Active Task-Inference-Guided Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/2001.09227/) (2020)
- [Courteous Autonomous Cars](https://mlsanity.com/paper/1808.02633/) (2018)
- [Predicting Human Behavior in Unrepeated, Simultaneous-Move Games](https://mlsanity.com/paper/1306.0918/) (2013)
- [Reinforcement Learning with Iterative Reasoning for Merging in Dense Traffic](https://mlsanity.com/paper/2005.11895/) (2020)
- [Game-theoretic Modeling of Traffic in Unsignalized Intersection Network for Autonomous Vehicle Control Verification and Validation](https://mlsanity.com/paper/1910.07141/) (2019)
- [Expressing Diverse Human Driving Behavior with Probabilistic Rewards and Online Inference](https://mlsanity.com/paper/2008.08812/) (2020)

## Built on by (most cited first, 6 of 6)

- [Deep Multiagent Reinforcement Learning: Challenges and Directions](https://mlsanity.com/paper/2106.15691/) (2021)
- [Theory of Mind as Intrinsic Motivation for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2307.01158/) (2023)
- [MetaMind: General and Cognitive World Models in Multi-Agent Systems by Meta-Theory of Mind](https://mlsanity.com/paper/2603.00808/) (2026)
- [Multi-Agent Inverse Reinforcement Learning: Suboptimal Demonstrations and Alternative Solution Concepts](https://mlsanity.com/paper/2109.01178/) (2021)
- [Decision-Making for Autonomous Vehicles with Interaction-Aware Behavioral Prediction and Social-Attention Neural Network](https://mlsanity.com/paper/2310.20148/) (2023)
- [ToP-ToM: Trust-aware Robot Policy with Theory of Mind](https://mlsanity.com/paper/2311.04397/) (2023)

## Abstract (arXiv)

Reward function, as an incentive representation that recognizes humans' agency and rationalizes humans' actions, is particularly appealing for modeling human behavior in human-robot interaction. Inverse Reinforcement Learning is an effective way to retrieve reward functions from demonstrations. However, it has always been challenging when applying it to multi-agent settings since the mutual influence between agents has to be appropriately modeled. To tackle this challenge, previous work either exploits equilibrium solution concepts by assuming humans as perfectly rational optimizers with unbounded intelligence or pre-assigns humans' interaction strategies a priori. In this work, we advocate that humans are bounded rational and have different intelligence levels when reasoning about others' decision-making process, and such an inherent and latent characteristic should be accounted for in reward learning algorithms. Hence, we exploit such insights from Theory-of-Mind and propose a new multi-agent Inverse Reinforcement Learning framework that reasons about humans' latent intelligence levels during learning. We validate our approach in both zero-sum and general-sum games with synthetic agents and illustrate a practical application to learning human drivers' reward functions from real driving data. We compare our approach with two baseline algorithms. The results show that by reasoning about humans' latent intelligence levels, the proposed approach has more flexibility and capability to retrieve reward functions that explain humans' driving behaviors better.