mlsanityData to 7 Oct 2026

A Bayesian Approach to Robust Inverse Reinforcement Learning

arXiv 2309.08571 ↗First posted 15 Sep 2023cs.LGcited by 9

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

We consider a Bayesian approach to offline model-based inverse reinforcement learning (IRL). The proposed framework differs from existing offline model-based IRL approaches by performing simultaneous estimation of the expert's reward function and subjective model of environment dynamics. We make use of a class of prior distributions which parameterizes how accurate the expert's model of the environment is to develop efficient algorithms to estimate the expert's reward and subjective dynamics in high-dimensional settings. Our analysis reveals a novel insight that the estimated policy exhibits robust performance when the expert is believed (a priori) to have a highly accurate model of the environment. We verify this observation in the MuJoCo environments and show that our algorithms outperform state-of-the-art offline IRL algorithms.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 42 of its 68 references that are in the map. First posted on arXiv 15 Sep 2023.

Builds on

42 of 68 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learning9 Feb
  2. 2022Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees4 Oct
  3. 2022Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guarantees4 Oct
  4. 2022A Primer on Maximum Causal Entropy Inverse Reinforcement Learning22 Mar
  5. 2021Scalable Bayesian Inverse Reinforcement Learning12 Feb
  6. 2018Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior21 May

Other references · 36

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  6. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  7. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  8. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  9. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  10. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  11. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  12. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
24 more
  1. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  2. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
  3. 2017A unified view of entropy-regularized Markov decision processescited by 174
  4. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
  5. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  6. 2020Improved Contrastive Divergence Training of Energy Based Modelscited by 127
  7. 2019Imitation Learning as f-Divergence Minimizationcited by 121
  8. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  9. 2017Imitating Driver Behavior with Generative Adversarial Networkscited by 92
  10. 2018Lipschitz Continuity in Model-based Reinforcement Learningcited by 90
  11. 2020The Value Equivalence Principle for Model-Based Reinforcement Learningcited by 88
  12. 2020Objective Mismatch in Model-based Reinforcement Learningcited by 87
  13. 2021Feedback in Imitation Learning: The Three Regimes of Covariate Shiftcited by 68
  14. 2020Model-Based Inverse Reinforcement Learning from Visual Demonstrationscited by 52
  15. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learningcited by 50
  16. 2019On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inferencecited by 49
  17. 2017Occam's razor is insufficient to infer the preferences of irrational agentscited by 44
  18. 2021Visual Adversarial Imitation Learning using Variational Modelscited by 42
  19. 2020Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Modelscited by 27
  20. 2016Inverse Reinforcement Learning with Simultaneous Estimation of Rewards and Dynamicscited by 24
  21. 2019The Assistive Multi-Armed Banditcited by 19
  22. 2022Driving in Real Life with Inverse Reinforcement Learningcited by 18
  23. 2019Inverse Rational Control with Partially Observable Continuous Nonlinear Dynamicscited by 16
  24. 2023The Virtues of Laziness in Model-based RL: A Unified Objective and Algorithmscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 2; 2026: 4.

9 papers.

  1. 2024A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challengescited by 10
  2. 2023A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learningcited by 9
  3. 2024Value of Information and Reward Specification in Active Inference and POMDPscited by 4
  4. 2026Offline Policy Optimization with Posterior Samplingcited by 1
  5. 2025Exploring the Link Between Bayesian Inference and Embodied Intelligence: Toward Open Physical-World Embodied AI Systemscited by 0
  6. 2025Distributional Inverse Reinforcement Learningcited by 0
  7. 2026Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Beliefcited by 0
  8. 2026Provably Efficient Policy-Reward Co-Pretraining for Adversarial Imitation Learningcited by 0
  9. 2026Reading Cognition as Decisions Unfold in Words: A Factorized Inverse Decision Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning
  2. 2020Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimization
  3. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  4. 2025Offline Reinforcement Learning via Inverse Optimization
  5. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  6. 2025OffSim: Offline Simulator for Model-based Offline Inverse Reinforcement Learning
  7. 2025BiCQL-ML: A Bi-Level Conservative Q-Learning Framework for Maximum Likelihood Inverse Reinforcement Learning
  8. 2024The Virtues of Pessimism in Inverse Reinforcement Learning