# Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch

arXiv: https://arxiv.org/abs/2007.01174 · HTML: https://mlsanity.com/paper/2007.01174/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 29 of its 71 references that are in the map.
- First posted on arXiv 2 Jul 2020.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Reinforcement Learning in Healthcare: A Survey](https://mlsanity.com/paper/1908.08796/) (2019)
- [Challenges of Real-World Reinforcement Learning](https://mlsanity.com/paper/1904.12901/) (2019)
- [A Theory of Regularized Markov Decision Processes](https://mlsanity.com/paper/1901.11275/) (2019)
- [Generative Adversarial Imitation from Observation](https://mlsanity.com/paper/1807.06158/) (2018)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [Action Robust Reinforcement Learning and Applications in Continuous Control](https://mlsanity.com/paper/1901.09184/) (2019)
- [Provably Efficient Imitation Learning from Observation Alone](https://mlsanity.com/paper/1905.10948/) (2019)
- [Robust Reinforcement Learning for Continuous Control with Model Misspecification](https://mlsanity.com/paper/1906.07516/) (2019)
- [State Alignment-based Imitation Learning](https://mlsanity.com/paper/1911.10947/) (2019)
- [Robust Reinforcement Learning via Adversarial training with Langevin Dynamics](https://mlsanity.com/paper/2002.06063/) (2020)
- [Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior](https://mlsanity.com/paper/1805.08010/) (2018)
- [State-only Imitation with Transition Dynamics Mismatch](https://mlsanity.com/paper/2002.11879/) (2020)
- [Inverse Reinforcement Learning with Simultaneous Estimation of Rewards and Dynamics](https://mlsanity.com/paper/1604.03912/) (2016)
- [Interactive Teaching Algorithms for Inverse Reinforcement Learning](https://mlsanity.com/paper/1905.11867/) (2019)
- [Dual Policy Iteration](https://mlsanity.com/paper/1805.10755/) (2018)
- [Teaching Inverse Reinforcement Learners via Features and Demonstrations](https://mlsanity.com/paper/1810.08926/) (2018)
- [Learner-aware Teaching: Inverse Reinforcement Learning with Preferences and Constraints](https://mlsanity.com/paper/1906.00429/) (2019)
- [Balancing Two-Player Stochastic Games with Soft Q-Learning](https://mlsanity.com/paper/1802.03216/) (2018)
- [Deep Robust Kalman Filter](https://mlsanity.com/paper/1703.02310/) (2017)

## Built on by (most cited first, 30 of 33)

- [Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems](https://mlsanity.com/paper/2405.06624/) (2024)
- [Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees](https://mlsanity.com/paper/2210.01808/) (2022)
- [Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning](https://mlsanity.com/paper/2306.12755/) (2023)
- [Robust Imitation Learning against Variations in Environment Dynamics](https://mlsanity.com/paper/2206.09314/) (2022)
- [Cross-Domain Policy Adaptation by Capturing Representation Mismatch](https://mlsanity.com/paper/2405.15369/) (2024)
- [ODRL: A Benchmark for Off-Dynamics Reinforcement Learning](https://mlsanity.com/paper/2410.20750/) (2024)
- [Identifiability and generalizability from multiple experts in Inverse Reinforcement Learning](https://mlsanity.com/paper/2209.10974/) (2022)
- [Policy Gradient Algorithms for Robust MDPs with Non-Rectangular Uncertainty Sets](https://mlsanity.com/paper/2305.19004/) (2023)
- [Identifiability and Generalizability in Constrained Inverse Reinforcement Learning](https://mlsanity.com/paper/2306.00629/) (2023)
- [Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning](https://mlsanity.com/paper/2110.12997/) (2021)
- [Imitation Learning in Discounted Linear MDPs without exploration assumptions](https://mlsanity.com/paper/2405.02181/) (2024)
- [Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guarantees](https://mlsanity.com/paper/2210.01282/) (2022)
- [A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challenges](https://mlsanity.com/paper/2409.07569/) (2024)
- [Robust Learning from Observation with Model Misspecification](https://mlsanity.com/paper/2202.06003/) (2022)
- [Concept Alignment as a Prerequisite for Value Alignment](https://mlsanity.com/paper/2310.20059/) (2023)
- [OMPO: A Unified Framework for RL under Policy and Dynamics Shifts](https://mlsanity.com/paper/2405.19080/) (2024)
- [IL-SOAR : Imitation Learning with Soft Optimistic Actor cRitic](https://mlsanity.com/paper/2502.19859/) (2025)
- [Robust Entropy-regularized Markov Decision Processes](https://mlsanity.com/paper/2112.15364/) (2021)
- [Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification](https://mlsanity.com/paper/2403.06854/) (2024)
- [Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Duality](https://mlsanity.com/paper/2508.17448/) (2025)
- [Enhancing Inverse Reinforcement Learning through Encoding Dynamic Information in Reward Shaping](https://mlsanity.com/paper/2410.03847/) (2024)
- [Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment](https://mlsanity.com/paper/2410.23680/) (2024)
- [Learning Causally Invariant Reward Functions from Diverse Demonstrations](https://mlsanity.com/paper/2409.08012/) (2024)
- [Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gains](https://mlsanity.com/paper/2602.00603/) (2026)
- [Inversely Learning Transferable Rewards via Abstracted States](https://mlsanity.com/paper/2501.01669/) (2025)
- [Statistical Scenario Modelling and Lookalike Distributions for Multi-Variate AI Risk](https://mlsanity.com/paper/2502.14491/) (2025)
- [Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroids](https://mlsanity.com/paper/2509.12010/) (2025)
- [Balance Equation-based Distributionally Robust Offline Imitation Learning](https://mlsanity.com/paper/2511.07942/) (2025)
- [User-Oriented Robust Reinforcement Learning](https://mlsanity.com/paper/2202.07301/) (2022)
- [A Novel Variational Lower Bound for Inverse Reinforcement Learning](https://mlsanity.com/paper/2311.03698/) (2023)

## Abstract (arXiv)

We study the inverse reinforcement learning (IRL) problem under a transition dynamics mismatch between the expert and the learner. Specifically, we consider the Maximum Causal Entropy (MCE) IRL learner model and provide a tight upper bound on the learner's performance degradation based on the ℓ1-distance between the transition dynamics of the expert and the learner. Leveraging insights from the Robust RL literature, we propose a robust MCE IRL algorithm, which is a principled approach to help with this mismatch. Finally, we empirically demonstrate the stable performance of our algorithm compared to the standard MCE IRL algorithm under transition dynamics mismatches in both finite and continuous MDP problems.