mlsanityData to 7 Oct 2026

Scalable Bayesian Inverse Reinforcement Learning

arXiv 2102.06483 ↗First posted 12 Feb 2021cs.LGcited by 56

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Bayesian inference over the reward presents an ideal solution to the ill-posed nature of the inverse reinforcement learning problem. Unfortunately current methods generally do not scale well beyond the small tabular setting due to the need for an inner-loop MDP solver, and even non-Bayesian methods that do themselves scale often require extensive interaction with the environment to perform well, being inappropriate for high stakes or costly applications such as healthcare. In this paper we introduce our method, Approximate Variational Reward Imitation Learning (AVRIL), that addresses both of these issues by jointly learning an approximate posterior distribution over the reward that scales to arbitrarily complicated state spaces alongside an appropriate policy in a completely offline manner through a variational approach to said latent reward. Applying our method to real medical data alongside classic control simulations, we demonstrate Bayesian reward inference in environments beyond the scope of current methods, as well as task performance competitive with focused offline imitation learning algorithms.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 17 of its 41 references that are in the map. First posted on arXiv 12 Feb 2021.

Builds on

17 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2019Deep Bayesian Reward Learning from Preferences10 Dec

Other references · 16

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2016OpenAI Gymcited by 1,684
  6. 2016Generative Adversarial Imitation Learningcited by 1,576
  7. 2016Variational Inference: A Review for Statisticianscited by 1,466
  8. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  9. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 402
  10. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
  11. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
  12. 2020Strictly Batch Imitation Learning by Energy-based Distribution Matchingcited by 54
4 more
  1. 2020Bayesian Robust Optimization for Imitation Learningcited by 28
  2. 2019Truly Batch Apprenticeship Learning with Deep Successor Featurescited by 27
  3. 2016Inverse Reinforcement Learning with Simultaneous Estimation of Rewards and Dynamicscited by 24
  4. 2018Active Inverse Reward Designcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 11; 2023: 16; 2024: 18; 2025: 3; 2026: 5.

56 papers.

Sort:

By area:

  1. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challengescited by 145
  2. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  3. 2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 77
  4. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coveragecited by 26
  5. 2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learningcited by 26
  6. 2022Proximal Point Imitation Learningcited by 25
  7. 2023CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learningcited by 20
  8. 2022Benchmarking Constraint Inference in Inverse Reinforcement Learningcited by 18
  9. 2022POETREE: Interpretable Policy Learning with Adaptive Decision Treescited by 14
  10. 2021The Medkit-Learn(ing) Environment: Medical Decision Modelling through Simulationcited by 11
  11. 2022Teacher Forcing Recovers Reward Functions for Text Generationcited by 11
  12. 2023Massively Scalable Inverse Reinforcement Learning in Google Mapscited by 11
38 more
  1. 2024How to Leverage Diverse Demonstrations in Offline Imitation Learningcited by 11
  2. 2023A Bayesian Approach to Robust Inverse Reinforcement Learningcited by 9
  3. 2023Multi-intention Inverse Q-learning for Interpretable Behavior Representationcited by 9
  4. 2023Inverse Reinforcement Learning with the Average Reward Criterioncited by 8
  5. 2025Capturing Nuanced Preferences: Preference-Aligned Distillation for Small Language Modelscited by 8
  6. 2023Maximum Causal Entropy Inverse Constrained Reinforcement Learningcited by 7
  7. 2024In-Trajectory Inverse Reinforcement Learning: Learn Incrementally Before An Ongoing Trajectory Terminatescited by 7
  8. 2022Inverse Online Learning: Understanding Non-Stationary and Reactionary Policiescited by 6
  9. 2024Collaborative AI Teaming in Unknown Environments via Active Goal Deductioncited by 6
  10. 2024OLLIE: Imitation Learning from Offline Pretraining to Online Finetuningcited by 6
  11. 2024Grounded Answers for Multi-agent Decision-making Problem through Generative World Modelcited by 6
  12. 2023Curricular Subgoals for Inverse Reinforcement Learningcited by 5
  13. 2021Inverse Optimal Control Adapted to the Noise Characteristics of the Human Sensorimotor Systemcited by 4
  14. 2022Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experiencecited by 4
  15. 2022Environment Design for Inverse Reinforcement Learningcited by 4
  16. 2024Walking the Values in Bayesian Inverse Reinforcement Learningcited by 4
  17. 2024Factorised Active Inference for Strategic Multi-Agent Interactionscited by 4
  18. 2024Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignmentcited by 4
  19. 2022Practical Approaches for Fair Learning with Multitype and Multivariate Sensitive Attributescited by 3
  20. 2023Programmatic Imitation Learning from Unlabeled and Noisy Demonstrationscited by 3
  21. 2024Markov Balance Satisfaction Improves Performance in Strictly Batch Offline Imitation Learningcited by 3
  22. 2024LLMR: Knowledge Distillation with a Large Language Model-Induced Rewardcited by 3
  23. 2024Enhancing Inverse Reinforcement Learning through Encoding Dynamic Information in Reward Shapingcited by 3
  24. 2024Online Control-Informed Learningcited by 3
  25. 2022Synthetic Model Combination: An Instance-wise Approach to Unsupervised Ensemble Learningcited by 2
  26. 2023Conditional Kernel Imitation Learning for Continuous State Environmentscited by 2
  27. 2024RGMDT: Return-Gap-Minimizing Decision Tree Extraction in Non-Euclidean Metric Spacecited by 2
  28. 2026A Generalized Apprenticeship Learning Framework for Capturing Evolving Student Pedagogical Strategiescited by 2
  29. 2023Replicating Complex Dialogue Policy of Humans via Offline Imitation Learning with Supervised Regularizationcited by 1
  30. 2023Optimising Human-AI Collaboration by Learning Convincing Explanationscited by 1
  31. 2024Provably Efficient Exploration in Inverse Constrained Reinforcement Learningcited by 1
  32. 2025PAC Apprenticeship Learning with Bayesian Active Inverse Reinforcement Learningcited by 1
  33. 2022Model-based Offline Imitation Learning with Non-expert Datacited by 0
  34. 2022Reward Learning using Structural Motifs in Inverse Reinforcement Learningcited by 0
  35. 2023Kernel Density Bayesian Inverse Reinforcement Learningcited by 0
  36. 2023A Novel Variational Lower Bound for Inverse Reinforcement Learningcited by 0
  37. 2024Accelerating Inverse Reinforcement Learning with Expert Bootstrappingcited by 0
  38. 2024The Virtues of Pessimism in Inverse Reinforcement Learningcited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Nonparametric Bayesian Inverse Reinforcement Learning with Data-Parallel Gibbs Sampling
  2. 2024A Bayesian Solution To The Imitation Gap
  3. 2020Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimization
  4. 2024Bayesian Inverse Reinforcement Learning for Non-Markovian Rewards
  5. 2022A Hierarchical Bayesian Approach to Inverse Reinforcement Learning with Symbolic Reward Machines
  6. 2024How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approach
  7. 2025Reward Compatibility: A Framework for Inverse RL
  8. 2019Meta-Inverse Reinforcement Learning with Probabilistic Context Variables