# Learning Safe Policies with Expert Guidance

arXiv: https://arxiv.org/abs/1805.08313 · HTML: https://mlsanity.com/paper/1805.08313/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2018.

## Built on by (most cited first, 11 of 11)

- [Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences](https://mlsanity.com/paper/2002.09089/) (2020)
- [Bayesian Robust Optimization for Imitation Learning](https://mlsanity.com/paper/2007.12315/) (2020)
- [A Survey of Safe Reinforcement Learning and Constrained MDPs: A Technical Survey on Single-Agent and Multi-Agent Safety](https://mlsanity.com/paper/2505.17342/) (2025)
- [Policy Gradient Bayesian Robust Optimization for Imitation Learning](https://mlsanity.com/paper/2106.06499/) (2021)
- [Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planning](https://mlsanity.com/paper/2204.03597/) (2022)
- [How To Not Drive: Learning Driving Constraints from Demonstration](https://mlsanity.com/paper/2110.00645/) (2021)
- [Linearization and Identification of Multiple-Attractor Dynamical Systems through Laplacian Eigenmaps](https://mlsanity.com/paper/2202.09171/) (2022)
- [Is the Rush to Machine Learning Jeopardizing Safety? Results of a Survey](https://mlsanity.com/paper/2111.14324/) (2021)
- [Probabilistic Counterexample Guidance for Safer Reinforcement Learning (Extended Version)](https://mlsanity.com/paper/2307.04927/) (2023)
- [Robustness in the Face of Partial Identifiability in Reward Learning](https://mlsanity.com/paper/2501.06376/) (2025)
- [State-Augmented Graphs for Circular Economy Triage](https://mlsanity.com/paper/2512.15824/) (2025)

## Abstract (arXiv)

We propose a framework for ensuring safe behavior of a reinforcement learning agent when the reward function may be difficult to specify. In order to do this, we rely on the existence of demonstrations from expert policies, and we provide a theoretical framework for the agent to optimize in the space of rewards consistent with its existing knowledge. We propose two methods to solve the resulting optimization: an exact ellipsoid-based method and a method in the spirit of the "follow-the-perturbed-leader" algorithm. Our experiments demonstrate the behavior of our algorithm in both discrete and continuous problems. The trained agent safely avoids states with potential negative effects while imitating the behavior of the expert in the other states.