Data to 7 Oct 2026

Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences

arXiv 2002.09089 ↗First posted 21 Feb 2020cs.LG stat.MLcited by 73

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Bayesian reward learning from demonstrations enables rigorous safety and uncertainty analysis when performing imitation learning. However, Bayesian reward learning methods are typically computationally intractable for complex control problems. We propose Bayesian Reward Extrapolation (Bayesian REX), a highly efficient Bayesian reward learning algorithm that scales to high-dimensional imitation learning problems by pre-training a low-dimensional feature encoding via self-supervised tasks and then leveraging preferences over demonstrations to perform fast Bayesian inference. Bayesian REX can learn to play Atari games from demonstrations, without access to the game score and can generate 100,000 samples from the posterior over reward functions in only 5 minutes on a personal laptop. Bayesian REX also results in imitation learning performance that is competitive with or better than state-of-the-art methods that only learn point estimates of the reward function. Finally, Bayesian REX enables efficient high-confidence policy evaluation without having access to samples of the reward function. These high-confidence performance bounds can be used to rank the performance and risk of a variety of evaluation policies and provide a way to detect reward hacking behaviors.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 50 of its 87 references that are in the map. First posted on arXiv 21 Feb 2020.

Builds on

50 of 87 references.

Closest ancestors · 5

Same direction, newest first

2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
2019Causal Confusion in Imitation Learning28 May
2018Risk-Sensitive Generative Adversarial Imitation Learning13 Aug
2018EnsembleDAgger: A Bayesian Approach to Safe Imitation Learning22 Jul
2018Behavioral Cloning from Observation4 May

Other references · 45

2014Adam: A Method for Stochastic Optimizationcited by 36,960
2017Proximal Policy Optimization Algorithmscited by 14,441
2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
2017Deep reinforcement learning from human preferencescited by 3,846
2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
2016Concrete Problems in AI Safetycited by 1,582
2016Generative Adversarial Imitation Learningcited by 1,576
2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485

33 more

2016Tutorial on Variational Autoencoderscited by 404
2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
2016Successor Features for Transfer in Reinforcement Learningcited by 331
2018Reward learning from human preferences and demonstrations in Ataricited by 314
2016Cooperative Inverse Reinforcement Learningcited by 298
2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
2017Inverse Reward Designcited by 246
2017DART: Noise Injection for Robust Imitation Learningcited by 187
2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
2019Functional Variational Bayesian Neural Networkscited by 127
2017AI Safety Gridworldscited by 108
2014Optimizing the CVaR via Samplingcited by 98
2018Playing hard exploration games by watching YouTubecited by 98
2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 67
2019Learning Reward Functions by Integrating Human Demonstrations and Preferencescited by 64
2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 51
2019Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPscited by 37
2017Pragmatic-Pedagogic Value Alignmentcited by 36
2018Importance Sampling Policy Evaluation with an Estimated Behavior Policycited by 34
2013On the Complexity of Solving Markov Decision Problemscited by 31
2019Risk-Aware Active Inverse Reinforcement Learningcited by 26
2018Variational Bayesian dropout: pitfalls and fixescited by 22
2018Impossibility and Uncertainty Theorems in AI Value Alignment (or why your AGI should not have a utility function)cited by 22
2017Should Robots be Obedient?cited by 20
2017Efficient Probabilistic Performance Bounds for Inverse Reinforcement Learningcited by 17
2017PixelGAN Autoencoderscited by 14
2017DropoutDAgger: A Bayesian Approach to Safe Imitation Learningcited by 14
2018Learning under Misspecified Objective Spacescited by 14
2018Learning Safe Policies with Expert Guidancecited by 11
2018Projected BNNs: Avoiding weight-space pathologies by learning latent representations of neural network weightscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 14; 2022: 14; 2023: 19; 2024: 10; 2025: 6; 2026: 8.

73 papers.

Sort:

By area:

2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challengescited by 145
2023Getting aligned on representational alignmentcited by 142
2021ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learningcited by 98
2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
2021Imitation Learning: Progress, Taxonomies and Challengescited by 56
2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 55
2022Models of human preference for learning reward functionscited by 54
2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learningcited by 52
2021LazyDAgger: Reducing Context Switching in Interactive Imitation Learningcited by 50

38 more

2021Self-Supervised Online Reward Shaping in Sparse-Reward Environmentscited by 34
2020Bayesian Robust Optimization for Imitation Learningcited by 28
2021What are you optimizing for? Aligning Recommender Systems with Human Valuescited by 28
2020Robust Imitation Learning from Noisy Demonstrationscited by 26
2022The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Typescited by 26
2022Proximal Point Imitation Learningcited by 25
2024Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learningcited by 21
2023Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Modelscited by 20
2022A Ranking Game for Imitation Learningcited by 18
2021Offline Preference-Based Apprenticeship Learningcited by 17
2022Inducing Structure in Reward Learning by Learning Featurescited by 16
2022Identifiability and generalizability from multiple experts in Inverse Reinforcement Learningcited by 16
2023Rating-based Reinforcement Learningcited by 14
2023What Matters to You? Towards Visual Representation Alignment for Robot Learningcited by 14
2023SIRL: Similarity-based Implicit Representation Learningcited by 13
2021Inverse Contextual Bandits: Learning How Behavior Evolves over Timecited by 12
2021Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Taskscited by 12
2021Policy Gradient Bayesian Robust Optimization for Imitation Learningcited by 11
2023Aligning Robot and Human Representationscited by 11
2023Human-oriented Representation Learning for Robotic Manipulationcited by 10
2023Inverse Decision Modeling: Learning Interpretable Representations of Behaviorcited by 10
2021Efficient Performance Bounds for Primal-Dual Reinforcement Learning from Demonstrationscited by 9
2024Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewardscited by 9
2024Pareto-Optimal Learning from Preferences with Hidden Contextcited by 8
2024MAPLE: A Framework for Active Preference Learning Guided by Large Language Modelscited by 6
2021SOPE: Spectrum of Off-Policy Estimatorscited by 5
2023Langevin Thompson Sampling with Logarithmic Communication: Bandits and Reinforcement Learningcited by 5
2024RoboMD: Uncovering Robot Vulnerabilities through Semantic Potential Fieldscited by 5
2021Deception in Social Learning: A Multi-Agent Reinforcement Learning Perspectivecited by 4
2025Masked IRL: LLM-Guided Reward Disambiguation from Demonstrations and Languagecited by 4
2022Safe Inverse Reinforcement Learning via Control Barrier Functioncited by 3
2024Preference Elicitation for Offline Reinforcement Learningcited by 3
2025Contrastive Learning from Exploratory Actions: Leveraging Natural Interactions for Preference Elicitationcited by 3
2022Fail-Safe Adversarial Generative Imitation Learningcited by 2
2022Learning from Physical Human Feedback: An Object-Centric One-Shot Adaptation Methodcited by 2
2023DIP-RL: Demonstration-Inferred Preference Learning in Minecraftcited by 2
2023COPR: Continual Learning Human Preference through Optimal Policy Regularizationcited by 2
2024How Much Progress Did I Make? An Unexplored Human Feedback Signal for Teaching Robotscited by 2

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way