# Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior

arXiv: https://arxiv.org/abs/1805.08010 · HTML: https://mlsanity.com/paper/1805.08010/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- First posted on arXiv 21 May 2018.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Built on by (most cited first, 30 of 40)

- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Building Machines that Learn and Think with People](https://mlsanity.com/paper/2408.03943/) (2024)
- [LazyDAgger: Reducing Context Switching in Interactive Imitation Learning](https://mlsanity.com/paper/2104.00053/) (2021)
- [Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizability](https://mlsanity.com/paper/2209.08025/) (2022)
- [Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch](https://mlsanity.com/paper/2007.01174/) (2020)
- [SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards](https://mlsanity.com/paper/1905.11108/) (2019)
- [Human irrationality: both bad and good for reward inference](https://mlsanity.com/paper/2111.06956/) (2021)
- [On the Sensitivity of Reward Inference to Misspecified Human Models](https://mlsanity.com/paper/2212.04717/) (2022)
- [Modeling the Mistakes of Boundedly Rational Agents Within a Bayesian Theory of Mind](https://mlsanity.com/paper/2106.13249/) (2021)
- [Choice Set Misspecification in Reward Inference](https://mlsanity.com/paper/2101.07691/) (2021)
- [Inverse Contextual Bandits: Learning How Behavior Evolves over Time](https://mlsanity.com/paper/2107.06317/) (2021)
- [LLM-driven Imitation of Subrational Behavior : Illusion or Reality?](https://mlsanity.com/paper/2402.08755/) (2024)
- [Towards Modeling and Influencing the Dynamics of Human Learning](https://mlsanity.com/paper/2301.00901/) (2023)
- [Inverse Decision Modeling: Learning Interpretable Representations of Behavior](https://mlsanity.com/paper/2310.18591/) (2023)
- [A Bayesian Approach to Robust Inverse Reinforcement Learning](https://mlsanity.com/paper/2309.08571/) (2023)
- [First Contact: Unsupervised Human-Machine Co-Adaptation via Mutual Information Maximization](https://mlsanity.com/paper/2205.12381/) (2022)
- [MIRROR: Differentiable Deep Social Projection for Assistive Human-Robot Communication](https://mlsanity.com/paper/2203.02877/) (2022)
- [Toward AI Assistants That Let Designers Design](https://mlsanity.com/paper/2107.13074/) (2021)
- [A Hybrid Rule-Based and Data-Driven Approach to Driver Modeling through Particle Filtering](https://mlsanity.com/paper/2108.12820/) (2021)
- [A Unifying Bayesian Formulation of Measures of Interpretability in Human-AI](https://mlsanity.com/paper/2104.10743/) (2021)
- [Inverse Optimal Control Adapted to the Noise Characteristics of the Human Sensorimotor System](https://mlsanity.com/paper/2110.11130/) (2021)
- [On Optimizing Interventions in Shared Autonomy](https://mlsanity.com/paper/2112.09169/) (2021)
- [Limited or Biased: Modeling Sub-Rational Human Investors in Financial Markets](https://mlsanity.com/paper/2210.08569/) (2022)
- [Modeling Mobile Health Users as Reinforcement Learning Agents](https://mlsanity.com/paper/2212.00863/) (2022)
- [Robot Navigation in Risky, Crowded Environments: Understanding Human Preferences](https://mlsanity.com/paper/2303.08284/) (2023)
- [Discovering User Types: Mapping User Traits by Task-Specific Behaviors in Reinforcement Learning](https://mlsanity.com/paper/2307.08169/) (2023)
- [A behavioural transformer for effective collaboration between a robot and a non-stationary human](https://mlsanity.com/paper/2307.13447/) (2023)
- [Adaptive Agent Architecture for Real-time Human-Agent Teaming](https://mlsanity.com/paper/2103.04439/) (2021)
- [Utility-based Adaptive Teaching Strategies using Bayesian Theory of Mind](https://mlsanity.com/paper/2309.17275/) (2023)
- [Expectation Alignment: Handling Reward Misspecification in the Presence of Expectation Mismatch](https://mlsanity.com/paper/2404.08791/) (2024)

## Abstract (arXiv)

Inferring intent from observed behavior has been studied extensively within the frameworks of Bayesian inverse planning and inverse reinforcement learning. These methods infer a goal or reward function that best explains the actions of the observed agent, typically a human demonstrator. Another agent can use this inferred intent to predict, imitate, or assist the human user. However, a central assumption in inverse reinforcement learning is that the demonstrator is close to optimal. While models of suboptimal behavior exist, they typically assume that suboptimal actions are the result of some type of random noise or a known cognitive bias, like temporal inconsistency. In this paper, we take an alternative approach, and model suboptimal behavior as the result of internal model misspecification: the reason that user actions might deviate from near-optimal actions is that the user has an incorrect set of beliefs about the rules -- the dynamics -- governing how actions affect the environment. Our insight is that while demonstrated actions may be suboptimal in the real world, they may actually be near-optimal with respect to the user's internal model of the dynamics. By estimating these internal beliefs from observed behavior, we arrive at a new method for inferring intent. We demonstrate in simulation and in a user study with 12 participants that this approach enables us to more accurately model human intent, and can be used in a variety of applications, including offering assistance in a shared autonomy framework and inferring human preferences.