mlsanityData to 7 Oct 2026

Bayesian Robust Optimization for Imitation Learning

arXiv 2007.12315 ↗First posted 24 Jul 2020cs.LG stat.MLcited by 28

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

One of the main challenges in imitation learning is determining what action an agent should take when outside the state distribution of the demonstrations. Inverse reinforcement learning (IRL) can enable generalization to new states by learning a parameterized reward function, but these approaches still face uncertainty over the true reward function and corresponding optimal policy. Existing safe imitation learning approaches based on IRL deal with this uncertainty using a maxmin framework that optimizes a policy under the assumption of an adversarial reward function, whereas risk-neutral IRL approaches either optimize a policy for the mean or MAP reward function. While completely ignoring risk can lead to overly aggressive and unsafe policies, optimizing in a fully adversarial sense is also problematic as it can lead to overly conservative policies that perform poorly in practice. To provide a bridge between these two extremes, we propose Bayesian Robust Optimization for Imitation Learning (BROIL). BROIL leverages Bayesian reward function inference and a user specific risk tolerance to efficiently optimize a robust policy that balances expected return and conditional value at risk. Our empirical results show that BROIL provides a natural way to interpolate between return-maximizing and risk-minimizing behaviors and outperforms existing risk-sensitive and risk-neutral inverse reinforcement learning algorithms. Code is available at https://github.com/dsbrown1331/broil.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 26 of its 69 references that are in the map. First posted on arXiv 24 Jul 2020.

Builds on

26 of 69 references.

Closest ancestors · 3

Same direction, newest first

  1. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences21 Feb
  2. 2018An Algorithmic Perspective on Imitation Learning16 Nov
  3. 2018Risk-Sensitive Generative Adversarial Imitation Learning13 Aug

Other references · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  5. 2017Inverse Reward Designcited by 246
  6. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
  7. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  8. 2014Optimizing the CVaR via Samplingcited by 98
  9. 2018Confounding-Robust Policy Improvementcited by 73
  10. 2020Active Preference-Based Gaussian Process Regression for Reward Learningcited by 70
  11. 2019Worst Cases Policy Gradientscited by 52
  12. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 51
11 more
  1. 2020Partial Policy Iteration for L1-Robust Markov Decision Processescited by 49
  2. 2018Soft-Robust Actor-Critic Policy-Gradientcited by 37
  3. 2019Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPscited by 37
  4. 2018Learning a Prior over Intent via Meta-Inverse Reinforcement Learningcited by 36
  5. 2019Deep Bayesian Reward Learning from Preferencescited by 28
  6. 2019Risk-Aware Active Inverse Reinforcement Learningcited by 26
  7. 2014Robust Sample Average Approximationcited by 25
  8. 2018Robust Markov Decision Process: Beyond Rectangularitycited by 24
  9. 2017Efficient Probabilistic Performance Bounds for Inverse Reinforcement Learningcited by 17
  10. 2019Epistemic Risk-Sensitive Reinforcement Learningcited by 17
  11. 2018Learning Safe Policies with Expert Guidancecited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 5; 2023: 5; 2024: 4; 2025: 6; 2026: 1.

28 papers.

Sort:

By area:

  1. 2022Adversarial Training for High-Stakes Reliabilitycited by 60
  2. 2021Scalable Bayesian Inverse Reinforcement Learningcited by 56
  3. 2021Universal Off-Policy Evaluationcited by 43
  4. 2023Improving Generalization of Alignment with Human Preferences through Group Invariant Learningcited by 25
  5. 2023Distributional Soft Actor-Critic with Three Refinementscited by 22
  6. 2021Offline Preference-Based Apprenticeship Learningcited by 17
  7. 2022STAP: Sequencing Task-Agnostic Policiescited by 17
  8. 2024Partially Observable Task and Motion Planning with Uncertainty and Risk Awarenesscited by 17
  9. 2021Policy Gradient Bayesian Robust Optimization for Imitation Learningcited by 11
  10. 2021Efficient Performance Bounds for Primal-Dual Reinforcement Learning from Demonstrationscited by 9
  11. 2022On the convex formulations of robust Markov decision processescited by 8
  12. 2024Pareto-Optimal Learning from Preferences with Hidden Contextcited by 8
16 more
  1. 2021Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learningcited by 5
  2. 2021Fully General Online Imitation Learningcited by 4
  3. 2022RASR: Risk-Averse Soft-Robust MDPs with EVaR and Entropic Riskcited by 4
  4. 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
  5. 2022Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planningcited by 3
  6. 2025Recursive Entropic Risk Optimization in Discounted MDPs: Sample Complexity Bounds with a Generative Modelcited by 3
  7. 2024RL, but don't do anything I wouldn't docited by 2
  8. 2025Learning from Preferences and Mixed Demonstrations in General Settingscited by 1
  9. 2023Risk-Averse MDPs under Reward Ambiguitycited by 0
  10. 2023Bayesian Regret Minimization in Offline Banditscited by 0
  11. 2023Contextual Reliability: When Different Features Matter in Different Contextscited by 0
  12. 2024Adaptive Querying for Reward Learning from Human Feedbackcited by 0
  13. 2025Robustness in the Face of Partial Identifiability in Reward Learningcited by 0
  14. 2025Generalizable Image Repair for Robust Visual Controlcited by 0
  15. 2025Exploring the Link Between Bayesian Inference and Embodied Intelligence: Toward Open Physical-World Embodied AI Systemscited by 0
  16. 2026On the Sample Complexity of Discounted Reinforcement Learning with Optimized Certainty Equivalentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Bayesian Solution To The Imitation Gap
  2. 2020Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimization
  3. 2019A Divergence Minimization Perspective on Imitation Learning Methods
  4. 2026Adversarial Imitation Learning with General Function Approximation: Theoretical Analysis and Practical Algorithms
  5. 2026OSIL: Learning Offline Safe Imitation Policies with Safety Inferred from Non-preferred Trajectories
  6. 2022Bayesian Disturbance Injection: Robust Imitation Learning of Flexible Policies for Robot Manipulation
  7. 2021Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality
  8. 2023A Bayesian Approach to Robust Inverse Reinforcement Learning