mlsanityData to 7 Oct 2026

Walking the Values in Bayesian Inverse Reinforcement Learning

arXiv 2407.10971 ↗First posted 15 Jul 2024cs.LGcited by 4

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

The goal of Bayesian inverse reinforcement learning (IRL) is recovering a posterior distribution over reward functions using a set of demonstrations from an expert optimizing for a reward unknown to the learner. The resulting posterior over rewards can then be used to synthesize an apprentice policy that performs well on the same or a similar task. A key challenge in Bayesian IRL is bridging the computational gap between the hypothesis space of possible rewards and the likelihood, often defined in terms of Q values: vanilla Bayesian IRL needs to solve the costly forward planning problem - going from rewards to the Q values - at every step of the algorithm, which may need to be done thousands of times. We propose to solve this by a simple change: instead of focusing on primarily sampling in the space of rewards, we can focus on primarily working in the space of Q-values, since the computation required to go from Q-values to reward is radically cheaper. Furthermore, this reversion of the computation makes it easy to compute the gradient allowing efficient sampling using Hamiltonian Monte Carlo. We propose ValueWalk - a new Markov chain Monte Carlo method based on this insight - and illustrate its advantages on several tasks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 6 of its 23 references that are in the map. First posted on arXiv 15 Jul 2024.

Builds on

6 of 23 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Scalable Bayesian Inverse Reinforcement Learning12 Feb

Other references · 5

  1. 2018Pyro: Deep Universal Probabilistic Programmingcited by 379
  2. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challengescited by 145
  3. 2020Primal Wasserstein Imitation Learningcited by 120
  4. 2020Strictly Batch Imitation Learning by Energy-based Distribution Matchingcited by 54
  5. 2015Learning the Preferences of Ignorant, Inconsistent Agentscited by 45

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025PAC Apprenticeship Learning with Bayesian Active Inverse Reinforcement Learningcited by 1
  2. 2025Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroidscited by 1
  3. 2025Distributional Inverse Reinforcement Learningcited by 0
  4. 2026Q-based Variational Inverse Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Bayesian Inverse Reinforcement Learning for Non-Markovian Rewards
  2. 2020Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimization
  3. 2026Nonparametric Bayesian Inverse Reinforcement Learning with Data-Parallel Gibbs Sampling
  4. 2023Kernel Density Bayesian Inverse Reinforcement Learning
  5. 2023A Bayesian Approach to Robust Inverse Reinforcement Learning
  6. 2020Necessary and Sufficient Conditions for Inverse Reinforcement Learning of Bayesian Stopping Time Problems
  7. 2019Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning
  8. 2022A Hierarchical Bayesian Approach to Inverse Reinforcement Learning with Symbolic Reward Machines