mlsanityData to 7 Oct 2026

Bayesian Inverse Transition Learning for Offline Settings

arXiv 2308.05075 ↗First posted 9 Aug 2023cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline Reinforcement learning is commonly used for sequential decision-making in domains such as healthcare and education, where the rewards are known and the transition dynamics T must be estimated on the basis of batch data. A key challenge for all tasks is how to learn a reliable estimate of the transition dynamics T that produce near-optimal policies that are safe enough so that they never take actions that are far away from the best action with respect to their value functions and informative enough so that they communicate the uncertainties they have. Using data from an expert, we propose a new constraint-based approach that captures our desiderata for reliably learning a posterior distribution of the transition dynamics T that is free from gradients. Our results demonstrate that by using our constraints, we learn a high-performing policy, while considerably reducing the policy's variance over different datasets. We also explain how combining uncertainty estimation with these constraints can help us infer a partial ranking of actions that produce higher returns, and helps us infer safer and more informative policies for planning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 9 of its 20 references that are in the map. First posted on arXiv 9 Aug 2023.

Builds on

9 of 20 references.

References · 9

  1. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  2. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
  3. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
  4. 2018Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Searchcited by 81
  5. 2018Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behaviorcited by 40
  6. 2019Maximum Likelihood Constraint Inference for Inverse Reinforcement Learningcited by 33
  7. 2016Inverse Reinforcement Learning with Simultaneous Estimation of Rewards and Dynamicscited by 24
  8. 2022Learning Stochastic Dynamics with Statistics-Informed Neural Networkcited by 16
  9. 2021Identifying Decision Points for Safe and Interpretable Reinforcement Learning in Hypotension Treatmentcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization
  2. 2026Bayesian Conservative Policy Optimization (BCPO): A Novel Uncertainty-Calibrated Offline Reinforcement Learning with Credible Lower Bounds
  3. 2023A Bayesian Approach to Robust Inverse Reinforcement Learning
  4. 2026Offline Policy Optimization with Posterior Sampling
  5. 2023Contextual Conservative Q-Learning for Offline Reinforcement Learning
  6. 2026Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Belief
  7. 2025Beyond Non-Expert Demonstrations: Outcome-Driven Action Constraint for Offline Reinforcement Learning
  8. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning