Inverse Reinforcement Learning with Explicit Policy Estimates
On the map: Offline RL and imitation learning from data → Inverse reinforcement learning
Abstract (arXiv)
Various methods for solving the inverse reinforcement learning (IRL) problem have been developed independently in machine learning and economics. In particular, the method of Maximum Causal Entropy IRL is based on the perspective of entropy maximization, while related advances in the field of economics instead assume the existence of unobserved action shocks to explain expert behavior (Nested Fixed Point Algorithm, Conditional Choice Probability method, Nested Pseudo-Likelihood Algorithm). In this work, we make previously unknown connections between these related methods from both fields. We achieve this by showing that they all belong to a class of optimization problems, characterized by a common form of the objective, the associated policy and the objective gradient. We demonstrate key computational and algorithmic differences which arise between the methods due to an approximation of the optimal soft value function, and describe how this leads to more efficient algorithms. Using insights which emerge from our study of this class of optimization problems, we identify various problem scenarios and investigate each method's suitability for these problems.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 6 of its 28 references that are in the map. First posted on arXiv 4 Mar 2021.
Builds on
6 of 28 references.
References · 6
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
- 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
- 2020Quantifying Differences in Reward Functionscited by 62
- 2017Inverse Reinforcement Learning with Conditional Choice Probabilitiescited by 3
Built on by
Citing papers per year: 2022: 2; 2023: 0; 2024: 0; 2025: 1.
3 papers.
- 2022Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guaranteescited by 42
- 2022Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guaranteescited by 12
- 2025Analyzing sequential activity and travel decisions with interpretable deep inverse reinforcement learningcited by 0
- 2025Analyzing sequential activity and travel decisions with interpretable deep inverse reinforcement learning17 Mar 2025
- 2022Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees4 Oct 2022
- 2022Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guarantees4 Oct 2022
Similar, not cited
Close in wording, no citation either way
- 2019Generalized Maximum Causal Entropy for Inverse Reinforcement Learning
- 2020Revisiting Maximum Entropy Inverse Reinforcement Learning: New Perspectives and Algorithms
- 2022Weighted Maximum Entropy Inverse Reinforcement Learning
- 2022A Primer on Maximum Causal Entropy Inverse Reinforcement Learning
- 2026A Lecture Note on Offline RL and IRL, Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Models
- 2024Convergence of a model-free entropy-regularized inverse reinforcement learning algorithm
- 2021A Generalised Inverse Reinforcement Learning Framework
- 2025Statistical analysis of Inverse Entropy-regularized Reinforcement Learning