mlsanityData to 7 Oct 2026

Uniqueness and Complexity of Inverse MDP Models

arXiv 2206.01192 ↗First posted 2 Jun 2022cs.LG cs.CCcited by 2

Not on the map. Why?

Abstract (arXiv)

What is the action sequence aa'a" that was likely responsible for reaching state s"' (from state s) in 3 steps? Addressing such questions is important in causal reasoning and in reinforcement learning. Inverse "MDP" models p(aa'a"|ss"') can be used to answer them. In the traditional "forward" view, transition "matrix" p(s'|sa) and policy π(a|s) uniquely determine "everything": the whole dynamics p(as'a's"a"...|s), and with it, the action-conditional state process p(s's"...|saa'a"), the multi-step inverse models p(aa'a"...|ssi), etc. If the latter is our primary concern, a natural question, analogous to the forward case is to which extent 1-step inverse model p(a|ss') plus policy π(a|s) determine the multi-step inverse models or even the whole dynamics. In other words, can forward models be inferred from inverse models or even be side-stepped. This work addresses this question and variations thereof, and also whether there are efficient decision/inference algorithms for this.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 12 of its 25 references that are in the map. First posted on arXiv 2 Jun 2022.

Builds on

12 of 25 references.

References · 12

  1. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  2. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  3. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  4. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  5. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  6. 2016Variational Intrinsic Controlcited by 262
  7. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 142
  8. 2019Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learningcited by 114
  9. 2021Shaking the foundations: delusions in sequence models for interaction and controlcited by 61
  10. 2021Provable RL with Exogenous Distractors via Multistep Inverse Dynamicscited by 28
  11. 2021Domain Adversarial Reinforcement Learningcited by 19
  12. 2013SPUDD: Stochastic Planning using Decision Diagramscited by 7

Built on by

Cited by year

Citing papers per year: 2022: 2.

2 papers.

  1. 2022Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Modelscited by 42
  2. 2022Agent-Controller Representations: Principled Offline RL with Rich Exogenous Informationcited by 15

Similar, not cited

Close in wording, no citation either way

  1. 2018Inverse Reinforcement Learning via Nonparametric Spatio-Temporal Subgoal Modeling
  2. 2017Inverse Reinforcement Learning from Summary Data
  3. 2019Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning
  4. 2024Towards Generalized Inverse Reinforcement Learning
  5. 2018Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior
  6. 2022Identifiability and generalizability from multiple experts in Inverse Reinforcement Learning
  7. 2022Task-Guided IRL in POMDPs that Scales
  8. 2025Learning Reward Machines from Partially Observed Policies