mlsanityData to 14 Oct 2026

When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning

arXiv 2302.07457 ↗First posted 15 Feb 2023cs.LG cs.AIcited by 26

On the map: World models → Latent dynamics models and model-based RL

also in 1 more
Abstract (arXiv)

Offline inverse reinforcement learning (Offline IRL) aims to recover the structure of rewards and environment dynamics that underlie observed actions in a fixed, finite set of demonstrations from an expert agent. Accurate models of expertise in executing a task has applications in safety-sensitive applications such as clinical decision making and autonomous driving. However, the structure of an expert's preferences implicit in observed actions is closely linked to the expert's model of the environment dynamics (i.e. the ``world'' model). Thus, inaccurate models of the world obtained from finite data with limited coverage could compound inaccuracy in estimated rewards. To address this issue, we propose a bi-level optimization formulation of the estimation task wherein the upper level is likelihood maximization based upon a conservative model of the expert's policy (lower level). The policy model is conservative in that it maximizes reward subject to a penalty that is increasing in the uncertainty of the estimated model of the world. We propose a new algorithmic framework to solve the bi-level optimization problem formulation and provide statistical and computational guarantees of performance for the associated optimal reward estimator. Finally, we demonstrate that the proposed algorithm outperforms the state-of-the-art offline IRL and imitation learning benchmarks by a large margin, over the continuous control tasks in MuJoCo and different datasets in the D4RL benchmark.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 49 of its 71 references that are in the map. First posted on arXiv 15 Feb 2023.

Builds on

49 of 71 references.

References · 49

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  7. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  8. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  9. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  10. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  11. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  12. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
37 more
  1. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  2. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  3. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 229
  4. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 193
  5. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  6. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 183
  7. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 176
  8. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  9. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 164
  10. 2020A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Criticcited by 144
  11. 2020A Finite Time Analysis of Two Time-Scale Actor Critic Methodscited by 122
  12. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  13. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  14. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 95
  15. 2020Error Bounds of Imitating Policies and Environmentscited by 90
  16. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  17. 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Followingcited by 67
  18. 2021Scalable Bayesian Inverse Reinforcement Learningcited by 56
  19. 2021Identifiability in inverse reinforcement learningcited by 55
  20. 2020Strictly Batch Imitation Learning by Energy-based Distribution Matchingcited by 54
  21. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learningcited by 50
  22. 2021Visual Adversarial Imitation Learning using Variational Modelscited by 42
  23. 2022Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guaranteescited by 42
  24. 2020Efficient Sampling-Based Maximum Entropy Inverse Reinforcement Learning with Application to Autonomous Drivingcited by 34
  25. 2021Online Apprenticeship Learningcited by 29
  26. 2020On Computation and Generalization of Generative Adversarial Imitation Learningcited by 28
  27. 2019Truly Batch Apprenticeship Learning with Deep Successor Featurescited by 27
  28. 2016Inverse Reinforcement Learning with Simultaneous Estimation of Rewards and Dynamicscited by 24
  29. 2020Inverse Reinforcement Learning with Natural Language Goalscited by 20
  30. 2023CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learningcited by 20
  31. 2021Understanding the World Through Actioncited by 17
  32. 2022Identifiability and generalizability from multiple experts in Inverse Reinforcement Learningcited by 16
  33. 2022Offline Reinforcement Learning via Linear-Programming with Error-Bound Induced Constraintscited by 15
  34. 2020When Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergencecited by 13
  35. 2022Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guaranteescited by 12
  36. 2021Uncertainty Estimation Using Riemannian Model Dynamics for Offline Reinforcement Learningcited by 10
  37. 2022Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experiencecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 6; 2025: 10; 2026: 7.

26 papers.

Sort:

By area:

  1. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimismcited by 69
  2. 2025BLUR: A Bi-Level Optimization Approach for LLM Unlearningcited by 23
  3. 2024Learning from Imperfect Demonstrations with Self-Supervision for Robotic Manipulationcited by 14
  4. 2024How to Leverage Diverse Demonstrations in Offline Imitation Learningcited by 12
  5. 2025Optimistically Optimistic Exploration for Provably Efficient Infinite-Horizon Reinforcement and Imitation Learningcited by 12
  6. 2024A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challengescited by 10
  7. 2024Grounded Answers for Multi-agent Decision-making Problem through Generative World Modelcited by 6
  8. 2025Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Makingcited by 5
  9. 2025MisoDICE: Multi-Agent Imitation from Unlabeled Mixed-Quality Demonstrationscited by 3
  10. 2023Conditional Kernel Imitation Learning for Continuous State Environmentscited by 2
  11. 2024Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloningcited by 2
  12. 2024OMG-RL:Offline Model-based Guided Reward Learning for Heparin Treatmentcited by 1
14 more
  1. 2025Understanding Inverse Reinforcement Learning under Overparameterization: Non-Asymptotic Analysis and Global Optimalitycited by 1
  2. 2025Trust Region Reward Optimization and Proximal Inverse Reward Optimization Algorithmcited by 1
  3. 2025OffSim: Offline Simulator for Model-based Offline Inverse Reinforcement Learningcited by 1
  4. 2026Fast Rates for Inverse Reinforcement Learningcited by 1
  5. 2023FP-IRL: Fokker--Planck Inverse Reinforcement Learning -- A Physics-Constrained Approach to Markov Decision Processescited by 0
  6. 2025DO-IQS: Dynamics-Aware Offline Inverse Q-Learning for Optimal Stopping with Unknown Gain Functionscited by 0
  7. 2025Distributional Inverse Reinforcement Learningcited by 0
  8. 2025BiCQL-ML: A Bi-Level Conservative Q-Learning Framework for Maximum Likelihood Inverse Reinforcement Learningcited by 0
  9. 2026Inverse Contextual Bandits without Rewards: Learning from a Non-Stationary Learner via Suffix Imitationcited by 0
  10. 2026Interactive Inverse Reinforcement Learning of Interaction Scenarios via Bi-level Optimizationcited by 0
  11. 2026A Lecture Note on Offline RL and IRL, Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Modelscited by 0
  12. 2026Provably Efficient Policy-Reward Co-Pretraining for Adversarial Imitation Learningcited by 0
  13. 2026Language-Critique Imitation Learning from Suboptimal Demonstrationscited by 0
  14. 2026Loop-Free Inverse Reinforcement Learning via Sequential Value Recovery with Q-Score Matchingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A Bayesian Approach to Robust Inverse Reinforcement Learning
  2. 2024Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms
  3. 2024Inverse Reinforcement Learning with Sub-optimal Experts
  4. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  5. 2024The Virtues of Pessimism in Inverse Reinforcement Learning
  6. 2025Offline Reinforcement Learning via Inverse Optimization
  7. 2022Active Exploration for Inverse Reinforcement Learning
  8. 2022Environment Design for Inverse Reinforcement Learning