# Risk-Aware Active Inverse Reinforcement Learning

arXiv: https://arxiv.org/abs/1901.02161 · HTML: https://mlsanity.com/paper/1901.02161/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 7 of its 24 references that are in the map.
- First posted on arXiv 8 Jan 2019.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 7 of 7)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach](https://mlsanity.com/paper/1506.02188/) (2015)
- [Interactive Learning from Policy-Dependent Human Feedback](https://mlsanity.com/paper/1701.06049/) (2017)
- [Optimizing the CVaR via Sampling](https://mlsanity.com/paper/1404.3862/) (2014)
- [Efficient Probabilistic Performance Bounds for Inverse Reinforcement Learning](https://mlsanity.com/paper/1707.00724/) (2017)
- [On Ensuring that Intelligent Machines Are Well-Behaved](https://mlsanity.com/paper/1708.05448/) (2017)

## Built on by (most cited first, 26 of 26)

- [Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences](https://mlsanity.com/paper/2002.09089/) (2020)
- [Bayesian Robust Optimization for Imitation Learning](https://mlsanity.com/paper/2007.12315/) (2020)
- [Active Exploration for Inverse Reinforcement Learning](https://mlsanity.com/paper/2207.08645/) (2022)
- [Learner-aware Teaching: Inverse Reinforcement Learning with Preferences and Constraints](https://mlsanity.com/paper/1906.00429/) (2019)
- [Inducing Structure in Reward Learning by Learning Features](https://mlsanity.com/paper/2201.07082/) (2022)
- [End-to-End Imitation Learning with Safety Guarantees using Control Barrier Functions](https://mlsanity.com/paper/2212.11365/) (2022)
- [Time-Efficient Reward Learning via Visually Assisted Cluster Ranking](https://mlsanity.com/paper/2212.00169/) (2022)
- [BEDD: The MineRL BASALT Evaluation and Demonstrations Dataset for Training and Benchmarking Agents that Solve Fuzzy Tasks](https://mlsanity.com/paper/2312.02405/) (2023)
- [Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations](https://mlsanity.com/paper/1907.03976/) (2019)
- [Automatic Risk Adaptation in Distributional Reinforcement Learning](https://mlsanity.com/paper/2106.06317/) (2021)
- [Assisted Robust Reward Design](https://mlsanity.com/paper/2111.09884/) (2021)
- [Fully General Online Imitation Learning](https://mlsanity.com/paper/2102.08686/) (2021)
- [Environment Design for Inverse Reinforcement Learning](https://mlsanity.com/paper/2210.14972/) (2022)
- [Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planning](https://mlsanity.com/paper/2204.03597/) (2022)
- [Learning Formal Specifications from Membership and Preference Queries](https://mlsanity.com/paper/2307.10434/) (2023)
- [Learning from Physical Human Feedback: An Object-Centric One-Shot Adaptation Method](https://mlsanity.com/paper/2203.04951/) (2022)
- [Reward Learning from Suboptimal Demonstrations with Applications in Surgical Electrocautery](https://mlsanity.com/paper/2404.07185/) (2024)
- [RL, but don't do anything I wouldn't do](https://mlsanity.com/paper/2410.06213/) (2024)
- [Neural Task Success Classifiers for Robotic Manipulation from Few Real Demonstrations](https://mlsanity.com/paper/2107.00722/) (2021)
- [PAC Apprenticeship Learning with Bayesian Active Inverse Reinforcement Learning](https://mlsanity.com/paper/2508.03693/) (2025)
- [Preventing Imitation Learning with Adversarial Policy Ensembles](https://mlsanity.com/paper/2002.01059/) (2020)
- [MSVIPER: Improved Policy Distillation for Reinforcement-Learning-Based Robot Navigation](https://mlsanity.com/paper/2209.09079/) (2022)
- [Imitating careful experts to avoid catastrophic events](https://mlsanity.com/paper/2302.01193/) (2023)
- [Interactively Teaching an Inverse Reinforcement Learner with Limited Feedback](https://mlsanity.com/paper/2309.09095/) (2023)
- [Learning Safe Agent Behaviour from Human Preferences and Justifications via World Models](https://mlsanity.com/paper/2607.13172/) (2026)
- [Q-based Variational Inverse Reinforcement Learning](https://mlsanity.com/paper/2608.16888/) (2026)

## Abstract (arXiv)

Active learning from demonstration allows a robot to query a human for specific types of input to achieve efficient learning. Existing work has explored a variety of active query strategies; however, to our knowledge, none of these strategies directly minimize the performance risk of the policy the robot is learning. Utilizing recent advances in performance bounds for inverse reinforcement learning, we propose a risk-aware active inverse reinforcement learning algorithm that focuses active queries on areas of the state space with the potential for large generalization error. We show that risk-aware active learning outperforms standard active IRL approaches on gridworld, simulated driving, and table setting tasks, while also providing a performance-based stopping criterion that allows a robot to know when it has received enough demonstrations to safely perform a task.