# Inverse Reinforcement Learning with Simultaneous Estimation of Rewards and Dynamics

arXiv: https://arxiv.org/abs/1604.03912 · HTML: https://mlsanity.com/paper/1604.03912/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Apr 2016.

## Built on by (most cited first, 24 of 24)

- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [Scalable Bayesian Inverse Reinforcement Learning](https://mlsanity.com/paper/2102.06483/) (2021)
- [Strictly Batch Imitation Learning by Energy-based Distribution Matching](https://mlsanity.com/paper/2006.14154/) (2020)
- [Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch](https://mlsanity.com/paper/2007.01174/) (2020)
- [When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning](https://mlsanity.com/paper/2302.07457/) (2023)
- [CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learning](https://mlsanity.com/paper/2302.04782/) (2023)
- [How to Leverage Diverse Demonstrations in Offline Imitation Learning](https://mlsanity.com/paper/2405.17476/) (2024)
- [Inverse Decision Modeling: Learning Interpretable Representations of Behavior](https://mlsanity.com/paper/2310.18591/) (2023)
- [A Bayesian Approach to Robust Inverse Reinforcement Learning](https://mlsanity.com/paper/2309.08571/) (2023)
- [Performative Policy Gradient: Optimality in Performative Reinforcement Learning](https://mlsanity.com/paper/2512.20576/) (2025)
- [Explaining by Imitating: Understanding Decisions by Interpretable Policy Learning](https://mlsanity.com/paper/2310.19831/) (2023)
- [Enhancing Inverse Reinforcement Learning through Encoding Dynamic Information in Reward Shaping](https://mlsanity.com/paper/2410.03847/) (2024)
- [Necessary and Sufficient Conditions for Inverse Reinforcement Learning of Bayesian Stopping Time Problems](https://mlsanity.com/paper/2007.03481/) (2020)
- [Modeling Mobile Health Users as Reinforcement Learning Agents](https://mlsanity.com/paper/2212.00863/) (2022)
- [Model-Based Imitation Learning Using Entropy Regularization of Model and Policy](https://mlsanity.com/paper/2206.10101/) (2022)
- [AI Alignment Dialogues: An Interactive Approach to AI Alignment in Support Agents](https://mlsanity.com/paper/2301.06421/) (2023)
- [OMG-RL:Offline Model-based Guided Reward Learning for Heparin Treatment](https://mlsanity.com/paper/2409.13299/) (2024)
- [Inversely Learning Transferable Rewards via Abstracted States](https://mlsanity.com/paper/2501.01669/) (2025)
- [Model-Based Reinforcement Learning Under Confounding](https://mlsanity.com/paper/2512.07528/) (2025)
- [FP-IRL: Fokker--Planck Inverse Reinforcement Learning -- A Physics-Constrained Approach to Markov Decision Processes](https://mlsanity.com/paper/2306.10407/) (2023)
- [Bayesian Inverse Transition Learning for Offline Settings](https://mlsanity.com/paper/2308.05075/) (2023)
- [Bayesian Inverse Transition Learning: Learning Dynamics From Near-Optimal Trajectories](https://mlsanity.com/paper/2411.05174/) (2024)
- [DO-IQS: Dynamics-Aware Offline Inverse Q-Learning for Optimal Stopping with Unknown Gain Functions](https://mlsanity.com/paper/2503.03515/) (2025)
- [Safe Explicable Policy Search](https://mlsanity.com/paper/2503.07848/) (2025)

## Abstract (arXiv)

Inverse Reinforcement Learning (IRL) describes the problem of learning an unknown reward function of a Markov Decision Process (MDP) from observed behavior of an agent. Since the agent's behavior originates in its policy and MDP policies depend on both the stochastic system dynamics as well as the reward function, the solution of the inverse problem is significantly influenced by both. Current IRL approaches assume that if the transition model is unknown, additional samples from the system's dynamics are accessible, or the observed behavior provides enough samples of the system's dynamics to solve the inverse problem accurately. These assumptions are often not satisfied. To overcome this, we present a gradient-based IRL approach that simultaneously estimates the system's dynamics. By solving the combined optimization problem, our approach takes into account the bias of the demonstrations, which stems from the generating policy. The evaluation on a synthetic MDP and a transfer learning task shows improvements regarding the sample efficiency as well as the accuracy of the estimated reward functions and transition models.