mlsanityData to 14 Oct 2026

Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification

arXiv 2403.06854 ↗First posted 11 Mar 2024cs.LGcited by 4

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

also in 1 more
Abstract (arXiv)

Inverse reinforcement learning (IRL) aims to infer an agent's preferences (represented as a reward function R) from their behaviour (represented as a policy π). To do this, we need a behavioural model of how π relates to R. In the current literature, the most common behavioural models are optimality, Boltzmann-rationality, and causal entropy maximisation. However, the true relationship between a human's preferences and their behaviour is much more complex than any of these behavioural models. This means that the behavioural models are misspecified, which raises the concern that they may lead to systematic errors if applied to real data. In this paper, we analyse how sensitive the IRL problem is to misspecification of the behavioural model. Specifically, we provide necessary and sufficient conditions that completely characterise how the observed data may differ from the assumed behavioural model without incurring an error above a given threshold. In addition to this, we also characterise the conditions under which a behavioural model is robust to small perturbations of the observed policy, and we analyse how robust many behavioural models are to misspecification of their parameter values (such as e.g. the discount rate). Our analysis suggests that the IRL problem is highly sensitive to misspecification, in the sense that very mild misspecification can lead to very large errors in the inferred reward function.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 17 of its 29 references that are in the map. First posted on arXiv 11 Mar 2024.

Builds on

17 of 29 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Identifiability and Generalizability in Constrained Inverse Reinforcement Learning1 Jun
  2. 2021Identifiability in inverse reinforcement learning7 Jun
  3. 2020Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch2 Jul

Other references · 14

  1. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
  2. 2016Cooperative Inverse Reinforcement Learningcited by 300
  3. 2021What Matters for Adversarial Imitation Learning?cited by 70
  4. 2020Quantifying Differences in Reward Functionscited by 63
  5. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  6. 2017Occam's razor is insufficient to infer the preferences of irrational agentscited by 45
  7. 2018Penalizing side effects using stepwise relative reachabilitycited by 38
  8. 2020Understanding Learned Reward Functionscited by 37
  9. 2020Avoiding Side Effects By Considering Future Taskscited by 32
  10. 2022On the Sensitivity of Reward Inference to Misspecified Human Modelscited by 25
  11. 2020Avoiding Side Effects in Complex Environmentscited by 24
  12. 2022Dynamics-Aware Comparison of Learned Reward Functionscited by 17
2 more
  1. 2023STARC: A General Framework For Quantifying Differences Between Reward Functionscited by 15
  2. 2021Choice Set Misspecification in Reward Inferencecited by 13

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 96
  2. 2025Modeling Others' Minds as Codecited by 7
  3. 2024Provably Efficient Exploration in Inverse Constrained Reinforcement Learningcited by 1
  4. 2025Robustness in the Face of Partial Identifiability in Reward Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Misspecification in Inverse Reinforcement Learning
  2. 2024Partial Identifiability and Misspecification in Inverse Reinforcement Learning
  3. 2026Quantifying Potential Observation Missingness in Inverse Reinforcement Learning
  4. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  5. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  6. 2024Partial Identifiability in Inverse Reinforcement Learning For Agents With Non-Exponential Discounting
  7. 2020Accounting for Human Learning when Inferring Human Preferences
  8. 2017Inverse Reinforcement Learning from Summary Data