mlsanityData to 7 Oct 2026

Policy Gradient Bayesian Robust Optimization for Imitation Learning

arXiv 2106.06499 ↗First posted 11 Jun 2021cs.LG cs.AIcited by 11

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

The difficulty in specifying rewards for many real-world problems has led to an increased focus on learning rewards from human feedback, such as demonstrations. However, there are often many different reward functions that explain the human feedback, leaving agents with uncertainty over what the true reward function is. While most policy optimization approaches handle this uncertainty by optimizing for expected performance, many applications demand risk-averse behavior. We derive a novel policy gradient-style robust optimization approach, PG-BROIL, that optimizes a soft-robust objective that balances expected performance and risk. To the best of our knowledge, PG-BROIL is the first policy optimization algorithm robust to a distribution of reward hypotheses which can scale to continuous MDPs. Results suggest that PG-BROIL can produce a family of behaviors ranging from risk-neutral to risk-averse and outperforms state-of-the-art imitation learning algorithms when learning from ambiguous demonstrations by hedging against uncertainty, rather than seeking to uniquely identify the demonstrator's reward function.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 37 of its 66 references that are in the map. First posted on arXiv 11 Jun 2021.

Builds on

37 of 66 references.

Closest ancestors · 5

Same direction, newest first

  1. 2020Bayesian Robust Optimization for Imitation Learning24 Jul
  2. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences21 Feb
  3. 2018An Algorithmic Perspective on Imitation Learning16 Nov
  4. 2018Risk-Sensitive Generative Adversarial Imitation Learning13 Aug
  5. 2018Behavioral Cloning from Observation4 May

Other references · 32

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2016Concrete Problems in AI Safetycited by 1,582
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2017Constrained Policy Optimizationcited by 928
  9. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  10. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  11. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  12. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
20 more
  1. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  2. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  3. 2017Inverse Reward Designcited by 246
  4. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 156
  5. 2017A General Safety Framework for Learning-Based Control in Uncertain Robotic Systemscited by 150
  6. 2013Risk-sensitive Reinforcement Learningcited by 129
  7. 2014Optimizing the CVaR via Samplingcited by 98
  8. 2016Query-Efficient Imitation Learning for End-to-End Autonomous Drivingcited by 83
  9. 2019Worst Cases Policy Gradientscited by 52
  10. 2021LazyDAgger: Reducing Context Switching in Interactive Imitation Learningcited by 50
  11. 2018Soft-Robust Actor-Critic Policy-Gradientcited by 37
  12. 2018Learning a Prior over Intent via Meta-Inverse Reinforcement Learningcited by 36
  13. 2020Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learningcited by 28
  14. 2019Entropic Risk Measure in Policy Searchcited by 19
  15. 2017Efficient Probabilistic Performance Bounds for Inverse Reinforcement Learningcited by 17
  16. 2018Simplifying Reward Design through Divide-and-Conquercited by 12
  17. 2018Learning Safe Policies with Expert Guidancecited by 11
  18. 2020ABC-LMPC: Safe Sample-Based Learning MPC for Stochastic Nonlinear Dynamical Systems with Adjustable Boundary Conditionscited by 8
  19. 2020Entropic Risk Constrained Soft-Robust Policy Optimizationcited by 3
  20. 2020Soft-Robust Algorithms for Batch Reinforcement Learningcited by 3

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 6; 2023: 4.

11 papers.

  1. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  2. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 55
  3. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learningcited by 52
  4. 2023Improving Generalization of Alignment with Human Preferences through Group Invariant Learningcited by 25
  5. 2022A General Framework for quantifying Aleatoric and Epistemic uncertainty in Graph Neural Networkscited by 12
  6. 2022On the convex formulations of robust Markov decision processescited by 8
  7. 2022RASR: Risk-Averse Soft-Robust MDPs with EVaR and Entropic Riskcited by 4
  8. 2021Learning to Be Cautiouscited by 3
  9. 2022Fail-Safe Adversarial Generative Imitation Learningcited by 2
  10. 2023Bayesian Regret Minimization in Offline Banditscited by 0
  11. 2023Contextual Reliability: When Different Features Matter in Different Contextscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026OSIL: Learning Offline Safe Imitation Policies with Safety Inferred from Non-preferred Trajectories
  2. 2025Learning safe, constrained policies via imitation learning: Connection to Probabilistic Inference and a Naive Algorithm
  3. 2021Stochastic convex optimization for provably efficient apprenticeship learning
  4. 2022Bayesian Disturbance Injection: Robust Imitation Learning of Flexible Policies for Robot Manipulation
  5. 2025Imitation Learning as Return Distribution Matching
  6. 2021Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality
  7. 2019Deep Bayesian Reward Learning from Preferences
  8. 2024A Bayesian Solution To The Imitation Gap