# Bayesian Robust Optimization for Imitation Learning

arXiv: https://arxiv.org/abs/2007.12315 · HTML: https://mlsanity.com/paper/2007.12315/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 26 of its 69 references that are in the map.
- First posted on arXiv 24 Jul 2020.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Inverse Reward Design](https://mlsanity.com/paper/1711.02827/) (2017)
- [Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach](https://mlsanity.com/paper/1506.02188/) (2015)
- [Safe Policy Improvement with Baseline Bootstrapping](https://mlsanity.com/paper/1712.06924/) (2017)
- [Optimizing the CVaR via Sampling](https://mlsanity.com/paper/1404.3862/) (2014)
- [Confounding-Robust Policy Improvement](https://mlsanity.com/paper/1805.08593/) (2018)
- [Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences](https://mlsanity.com/paper/2002.09089/) (2020)
- [Active Preference-Based Gaussian Process Regression for Reward Learning](https://mlsanity.com/paper/2005.02575/) (2020)
- [Safe Policy Improvement by Minimizing Robust Baseline Regret](https://mlsanity.com/paper/1607.03842/) (2016)
- [Worst Cases Policy Gradients](https://mlsanity.com/paper/1911.03618/) (2019)
- [Partial Policy Iteration for L1-Robust Markov Decision Processes](https://mlsanity.com/paper/2006.09484/) (2020)
- [Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPs](https://mlsanity.com/paper/1902.07605/) (2019)
- [Soft-Robust Actor-Critic Policy-Gradient](https://mlsanity.com/paper/1803.04848/) (2018)
- [Learning a Prior over Intent via Meta-Inverse Reinforcement Learning](https://mlsanity.com/paper/1805.12573/) (2018)
- [Deep Bayesian Reward Learning from Preferences](https://mlsanity.com/paper/1912.04472/) (2019)
- [Risk-Aware Active Inverse Reinforcement Learning](https://mlsanity.com/paper/1901.02161/) (2019)
- [Robust Sample Average Approximation](https://mlsanity.com/paper/1408.4445/) (2014)
- [Robust Markov Decision Process: Beyond Rectangularity](https://mlsanity.com/paper/1811.00215/) (2018)
- [Efficient Probabilistic Performance Bounds for Inverse Reinforcement Learning](https://mlsanity.com/paper/1707.00724/) (2017)
- [Epistemic Risk-Sensitive Reinforcement Learning](https://mlsanity.com/paper/1906.06273/) (2019)
- [Risk-Sensitive Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1808.04468/) (2018)
- [Learning Safe Policies with Expert Guidance](https://mlsanity.com/paper/1805.08313/) (2018)

## Built on by (most cited first, 28 of 28)

- [Adversarial Training for High-Stakes Reliability](https://mlsanity.com/paper/2205.01663/) (2022)
- [Scalable Bayesian Inverse Reinforcement Learning](https://mlsanity.com/paper/2102.06483/) (2021)
- [Universal Off-Policy Evaluation](https://mlsanity.com/paper/2104.12820/) (2021)
- [Distributional Soft Actor-Critic with Three Refinements](https://mlsanity.com/paper/2310.05858/) (2023)
- [Improving Generalization of Alignment with Human Preferences through Group Invariant Learning](https://mlsanity.com/paper/2310.11971/) (2023)
- [STAP: Sequencing Task-Agnostic Policies](https://mlsanity.com/paper/2210.12250/) (2022)
- [Offline Preference-Based Apprenticeship Learning](https://mlsanity.com/paper/2107.09251/) (2021)
- [Partially Observable Task and Motion Planning with Uncertainty and Risk Awareness](https://mlsanity.com/paper/2403.10454/) (2024)
- [Policy Gradient Bayesian Robust Optimization for Imitation Learning](https://mlsanity.com/paper/2106.06499/) (2021)
- [Efficient Performance Bounds for Primal-Dual Reinforcement Learning from Demonstrations](https://mlsanity.com/paper/2112.14004/) (2021)
- [On the convex formulations of robust Markov decision processes](https://mlsanity.com/paper/2209.10187/) (2022)
- [Pareto-Optimal Learning from Preferences with Hidden Context](https://mlsanity.com/paper/2406.15599/) (2024)
- [Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learning](https://mlsanity.com/paper/2110.05286/) (2021)
- [Recursive Entropic Risk Optimization in Discounted MDPs: Sample Complexity Bounds with a Generative Model](https://mlsanity.com/paper/2506.00286/) (2025)
- [Fully General Online Imitation Learning](https://mlsanity.com/paper/2102.08686/) (2021)
- [RASR: Risk-Averse Soft-Robust MDPs with EVaR and Entropic Risk](https://mlsanity.com/paper/2209.04067/) (2022)
- [Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Duality](https://mlsanity.com/paper/2508.17448/) (2025)
- [Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planning](https://mlsanity.com/paper/2204.03597/) (2022)
- [RL, but don't do anything I wouldn't do](https://mlsanity.com/paper/2410.06213/) (2024)
- [Learning from Preferences and Mixed Demonstrations in General Settings](https://mlsanity.com/paper/2508.14027/) (2025)
- [On the Sample Complexity of Discounted Reinforcement Learning with Optimized Certainty Equivalents](https://mlsanity.com/paper/2605.21763/) (2026)
- [Risk-Averse MDPs under Reward Ambiguity](https://mlsanity.com/paper/2301.01045/) (2023)
- [Bayesian Regret Minimization in Offline Bandits](https://mlsanity.com/paper/2306.01237/) (2023)
- [Contextual Reliability: When Different Features Matter in Different Contexts](https://mlsanity.com/paper/2307.10026/) (2023)
- [Adaptive Querying for Reward Learning from Human Feedback](https://mlsanity.com/paper/2412.07990/) (2024)
- [Robustness in the Face of Partial Identifiability in Reward Learning](https://mlsanity.com/paper/2501.06376/) (2025)
- [Generalizable Image Repair for Robust Visual Control](https://mlsanity.com/paper/2503.05911/) (2025)
- [Exploring the Link Between Bayesian Inference and Embodied Intelligence: Toward Open Physical-World Embodied AI Systems](https://mlsanity.com/paper/2507.21589/) (2025)

## Abstract (arXiv)

One of the main challenges in imitation learning is determining what action an agent should take when outside the state distribution of the demonstrations. Inverse reinforcement learning (IRL) can enable generalization to new states by learning a parameterized reward function, but these approaches still face uncertainty over the true reward function and corresponding optimal policy. Existing safe imitation learning approaches based on IRL deal with this uncertainty using a maxmin framework that optimizes a policy under the assumption of an adversarial reward function, whereas risk-neutral IRL approaches either optimize a policy for the mean or MAP reward function. While completely ignoring risk can lead to overly aggressive and unsafe policies, optimizing in a fully adversarial sense is also problematic as it can lead to overly conservative policies that perform poorly in practice. To provide a bridge between these two extremes, we propose Bayesian Robust Optimization for Imitation Learning (BROIL). BROIL leverages Bayesian reward function inference and a user specific risk tolerance to efficiently optimize a robust policy that balances expected return and conditional value at risk. Our empirical results show that BROIL provides a natural way to interpolate between return-maximizing and risk-minimizing behaviors and outperforms existing risk-sensitive and risk-neutral inverse reinforcement learning algorithms. Code is available at https://github.com/dsbrown1331/broil.