mlsanityData to 14 Oct 2026

Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees

arXiv 2210.01808 ↗First posted 4 Oct 2022cs.LG stat.MLcited by 42

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Inverse reinforcement learning (IRL) aims to recover the reward function and the associated optimal policy that best fits observed sequences of states and actions implemented by an expert. Many algorithms for IRL have an inherently nested structure: the inner loop finds the optimal policy given parametrized rewards while the outer loop updates the estimates towards optimizing a measure of fit. For high dimensional environments such nested-loop structure entails a significant computational burden. To reduce the computational burden of a nested loop, novel methods such as SQIL [1] and IQ-Learn [2] emphasize policy estimation at the expense of reward estimation accuracy. However, without accurate estimated rewards, it is not possible to do counterfactual analysis such as predicting the optimal policy under different environment dynamics and/or learning new tasks. In this paper we develop a novel single-loop algorithm for IRL that does not compromise reward estimation accuracy. In the proposed algorithm, each policy improvement step is followed by a stochastic gradient step for likelihood maximization. We show that the proposed algorithm provably converges to a stationary solution with a finite-time guarantee. If the reward is parameterized linearly, we show the identified solution corresponds to the solution of the maximum entropy IRL problem. Finally, by using robotics control problems in MuJoCo and their transfer settings, we show that the proposed algorithm achieves superior performance compared with other IRL and imitation learning benchmarks.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 64 references that are in the map. First posted on arXiv 4 Oct 2022.

Builds on

31 of 64 references.

Closest ancestors · 5

Same direction, newest first

  1. 2021Efficient Performance Bounds for Primal-Dual Reinforcement Learning from Demonstrations28 Dec
  2. 2021Identifiability in inverse reinforcement learning7 Jun
  3. 2021Inverse Reinforcement Learning with Explicit Policy Estimates4 Mar
  4. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matching9 Nov
  5. 2020Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch2 Jul

Other references · 26

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2016Generative Adversarial Imitation Learningcited by 1,597
  3. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  4. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  5. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  6. 2018An Algorithmic Perspective on Imitation Learningcited by 459
  7. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  8. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  9. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 229
  10. 2020Bilevel Optimization: Convergence Analysis and Enhanced Designcited by 222
  11. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 193
  12. 2018Generative Adversarial Imitation from Observationcited by 184
14 more
  1. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 182
  2. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 176
  3. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  4. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 168
  5. 2020A Finite Time Analysis of Two Time-Scale Actor Critic Methodscited by 122
  6. 2021A Near-Optimal Algorithm for Stochastic Bilevel Optimization via Double-Momentumcited by 106
  7. 2019Finite-Sample Analysis for SARSA with Linear Function Approximationcited by 85
  8. 2019State Alignment-based Imitation Learningcited by 73
  9. 2019Random Expert Distillation: Imitation Learning via Expert Policy Support Estimationcited by 42
  10. 2020State-only Imitation with Transition Dynamics Mismatchcited by 37
  11. 2020On Computation and Generalization of Generative Adversarial Imitation Learningcited by 28
  12. 2018A Large-Scale Study on Regularization and Normalization in GANscited by 26
  13. 2020Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimizationcited by 22
  14. 2020When Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergencecited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 17; 2025: 12; 2026: 5.

42 papers.

Sort:

By area:

  1. 2024Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignmentcited by 32
  2. 2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learningcited by 26
  3. 2023Towards Theoretical Understanding of Inverse Reinforcement Learningcited by 21
  4. 2025Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignmentcited by 20
  5. 2023Identifiability and Generalizability in Constrained Inverse Reinforcement Learningcited by 16
  6. 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitionscited by 16
  7. 2024Imitation Learning in Discounted Linear MDPs without exploration assumptionscited by 15
  8. 2024Learning from Imperfect Demonstrations with Self-Supervision for Robotic Manipulationcited by 14
  9. 2024How to Leverage Diverse Demonstrations in Offline Imitation Learningcited by 12
  10. 2025On The Sample Complexity Bounds In Bilevel Reinforcement Learningcited by 10
  11. 2023A Bayesian Approach to Robust Inverse Reinforcement Learningcited by 9
  12. 2023Inverse Reinforcement Learning with the Average Reward Criterioncited by 8
30 more
  1. 2023Robust Visual Imitation Learning with Inverse Dynamics Representationscited by 8
  2. 2024In-Trajectory Inverse Reinforcement Learning: Learn Incrementally Before An Ongoing Trajectory Terminatescited by 7
  3. 2024OLLIE: Imitation Learning from Offline Pretraining to Online Finetuningcited by 6
  4. 2024Grounded Answers for Multi-agent Decision-making Problem through Generative World Modelcited by 6
  5. 2025Inverse Reinforcement Learning with Switching Rewards and History Dependency for Characterizing Animal Behaviorscited by 6
  6. 2023Infer and Adapt: Bipedal Locomotion Reward Learning from Demonstrations via Inverse Reinforcement Learningcited by 5
  7. 2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRiticcited by 5
  8. 2025Inverse Reinforcement Learning with Just Classification and a Few Regressionscited by 5
  9. 2024Inverse Reinforcement Learning with Sub-optimal Expertscited by 4
  10. 2024Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignmentcited by 4
  11. 2024Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Modelscited by 4
  12. 2025From Demonstrations to Rewards: Alignment Without Explicit Human Preferencescited by 4
  13. 2024Regularized Q-Learning with Linear Function Approximationcited by 2
  14. 2024Expert Proximity as Surrogate Rewards for Single Demonstration Imitation Learningcited by 2
  15. 2024Convergence of a model-free entropy-regularized inverse reinforcement learning algorithmcited by 2
  16. 2026Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gainscited by 2
  17. 2024Pareto Inverse Reinforcement Learning for Diverse Expert Policy Generationcited by 1
  18. 2024OMG-RL:Offline Model-based Guided Reward Learning for Heparin Treatmentcited by 1
  19. 2025Understanding Inverse Reinforcement Learning under Overparameterization: Non-Asymptotic Analysis and Global Optimalitycited by 1
  20. 2025Trust Region Reward Optimization and Proximal Inverse Reward Optimization Algorithmcited by 1
  21. 2024Inference of Utilities and Time Preference in Sequential Decision-Makingcited by 0
  22. 2024Inverse Reinforcement Learning with Multiple Planning Horizonscited by 0
  23. 2025Recursive Deep Inverse Reinforcement Learningcited by 0
  24. 2025DMRL: Data- and Model-aware Reward Learning for Data Extractioncited by 0
  25. 2025Distributional Inverse Reinforcement Learningcited by 0
  26. 2025From Data to Rewards: a Bilevel Optimization Perspective on Maximum Likelihood Estimationcited by 0
  27. 2026Multi-Objective Constraint Inference using Inverse reinforcement learningcited by 0
  28. 2026Interactive Inverse Reinforcement Learning of Interaction Scenarios via Bi-level Optimizationcited by 0
  29. 2026Efficient Hypergradient Descent for Inverse Reinforcement Learningcited by 0
  30. 2026Loop-Free Inverse Reinforcement Learning via Sequential Value Recovery with Q-Score Matchingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2019Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning
  2. 2022Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guarantees
  3. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  4. 2026Fast Rates for Inverse Reinforcement Learning
  5. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  6. 2021A Generalised Inverse Reinforcement Learning Framework
  7. 2026Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approach
  8. 2024How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approach