mlsanityData to 13 Oct 2026

Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior

arXiv 1805.08010 ↗First posted 21 May 2018cs.LG stat.MLcited by 40

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Inferring intent from observed behavior has been studied extensively within the frameworks of Bayesian inverse planning and inverse reinforcement learning. These methods infer a goal or reward function that best explains the actions of the observed agent, typically a human demonstrator. Another agent can use this inferred intent to predict, imitate, or assist the human user. However, a central assumption in inverse reinforcement learning is that the demonstrator is close to optimal. While models of suboptimal behavior exist, they typically assume that suboptimal actions are the result of some type of random noise or a known cognitive bias, like temporal inconsistency. In this paper, we take an alternative approach, and model suboptimal behavior as the result of internal model misspecification: the reason that user actions might deviate from near-optimal actions is that the user has an incorrect set of beliefs about the rules -- the dynamics -- governing how actions affect the environment. Our insight is that while demonstrated actions may be suboptimal in the real world, they may actually be near-optimal with respect to the user's internal model of the dynamics. By estimating these internal beliefs from observed behavior, we arrive at a new method for inferring intent. We demonstrate in simulation and in a user study with 12 participants that this approach enables us to more accurately model human intent, and can be used in a variety of applications, including offering assistance in a shared autonomy framework and inferring human preferences.

Key facts. Cited by 40 papers in the map (data through 13 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. First posted on arXiv 21 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2019: 1; 2020: 1; 2021: 11; 2022: 6; 2023: 9; 2024: 5; 2025: 4; 2026: 3.

40 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 751
  2. 2024Building Machines that Learn and Think with Peoplecited by 73
  3. 2021LazyDAgger: Reducing Context Switching in Interactive Imitation Learningcited by 50
  4. 2020Robust Inverse Reinforcement Learning under Transition Dynamics Mismatchcited by 33
  5. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
  6. 2019SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewardscited by 28
  7. 2021Human irrationality: both bad and good for reward inferencecited by 25
  8. 2022On the Sensitivity of Reward Inference to Misspecified Human Modelscited by 25
  9. 2021Modeling the Mistakes of Boundedly Rational Agents Within a Bayesian Theory of Mindcited by 19
  10. 2021Choice Set Misspecification in Reward Inferencecited by 13
  11. 2021Inverse Contextual Bandits: Learning How Behavior Evolves over Timecited by 12
  12. 2024LLM-driven Imitation of Subrational Behavior : Illusion or Reality?cited by 12
28 more
  1. 2023Towards Modeling and Influencing the Dynamics of Human Learningcited by 11
  2. 2023Inverse Decision Modeling: Learning Interpretable Representations of Behaviorcited by 10
  3. 2023A Bayesian Approach to Robust Inverse Reinforcement Learningcited by 9
  4. 2022First Contact: Unsupervised Human-Machine Co-Adaptation via Mutual Information Maximizationcited by 8
  5. 2022MIRROR: Differentiable Deep Social Projection for Assistive Human-Robot Communicationcited by 7
  6. 2021Toward AI Assistants That Let Designers Designcited by 6
  7. 2021A Hybrid Rule-Based and Data-Driven Approach to Driver Modeling through Particle Filteringcited by 6
  8. 2021A Unifying Bayesian Formulation of Measures of Interpretability in Human-AIcited by 5
  9. 2021Inverse Optimal Control Adapted to the Noise Characteristics of the Human Sensorimotor Systemcited by 4
  10. 2021On Optimizing Interventions in Shared Autonomycited by 3
  11. 2022Limited or Biased: Modeling Sub-Rational Human Investors in Financial Marketscited by 2
  12. 2022Modeling Mobile Health Users as Reinforcement Learning Agentscited by 2
  13. 2023Robot Navigation in Risky, Crowded Environments: Understanding Human Preferencescited by 2
  14. 2023Discovering User Types: Mapping User Traits by Task-Specific Behaviors in Reinforcement Learningcited by 2
  15. 2023A behavioural transformer for effective collaboration between a robot and a non-stationary humancited by 2
  16. 2021Adaptive Agent Architecture for Real-time Human-Agent Teamingcited by 1
  17. 2023Utility-based Adaptive Teaching Strategies using Bayesian Theory of Mindcited by 1
  18. 2024Expectation Alignment: Handling Reward Misspecification in the Presence of Expectation Mismatchcited by 1
  19. 2025Model-Based Reinforcement Learning Under Confoundingcited by 1
  20. 2023Bayesian Inverse Transition Learning for Offline Settingscited by 0
  21. 2024Choice Between Partial Trajectories: Disentangling Goals from Beliefscited by 0
  22. 2024Bayesian Inverse Transition Learning: Learning Dynamics From Near-Optimal Trajectoriescited by 0
  23. 2025Supervised Reward Inferencecited by 0
  24. 2025Safe Explicable Policy Searchcited by 0
  25. 2025A Descriptive and Normative Theory of Human Beliefs in RLHFcited by 0
  26. 2026Learning the Preferences of a Learning Agentcited by 0
  27. 2026Quantifying Potential Observation Missingness in Inverse Reinforcement Learningcited by 0
  28. 2026Watch, Infer, Coordinate: Inferring Robot Partner Constraints for Zero-Shot Coordinationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2018Inverse Rational Control: Inferring What You Think from How You Forage
  2. 2024Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification
  3. 2018Inverse Reinforcement Learning via Nonparametric Spatio-Temporal Subgoal Modeling
  4. 2017Inverse Reinforcement Learning from Summary Data
  5. 2019On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inference
  6. 2022Misspecification in Inverse Reinforcement Learning
  7. 2024Active Inference as a Model of Agency
  8. 2019Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning