# Safer Reinforcement Learning through Transferable Instinct Networks

arXiv: https://arxiv.org/abs/2107.06686 · HTML: https://mlsanity.com/paper/2107.06686/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 37 references that are in the map.
- First posted on arXiv 14 Jul 2021.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation](https://mlsanity.com/paper/1708.05144/) (2017)
- [Learning to be Safe: Deep RL with a Safety Critic](https://mlsanity.com/paper/2010.14603/) (2020)
- [Conservative Safety Critics for Exploration](https://mlsanity.com/paper/2010.14497/) (2020)
- [Benchmarking Reinforcement Learning Algorithms on Real-World Robots](https://mlsanity.com/paper/1809.07731/) (2018)
- [Horizon: Facebook's Open Source Applied Reinforcement Learning Platform](https://mlsanity.com/paper/1811.00260/) (2018)
- [Deep Learning for Video Game Playing](https://mlsanity.com/paper/1708.07902/) (2017)
- [Modular Deep Reinforcement Learning with Temporal Logic Specifications](https://mlsanity.com/paper/1909.11591/) (2019)
- [Generalizing from a few environments in safety-critical reinforcement learning](https://mlsanity.com/paper/1907.01475/) (2019)
- [SafeLife 1.0: Exploring Side Effects in Complex Environments](https://mlsanity.com/paper/1912.01217/) (2019)
- [Safety-Guided Deep Reinforcement Learning via Online Gaussian Process Estimation](https://mlsanity.com/paper/1903.02526/) (2019)
- [Safe Reinforcement Learning through Meta-learned Instincts](https://mlsanity.com/paper/2005.03233/) (2020)

## Built on by (most cited first, 1 of 1)

- [A Domain-Agnostic Approach for Characterization of Lifelong Learning Systems](https://mlsanity.com/paper/2301.07799/) (2023)

## Abstract (arXiv)

Random exploration is one of the main mechanisms through which reinforcement learning (RL) finds well-performing policies. However, it can lead to undesirable or catastrophic outcomes when learning online in safety-critical environments. In fact, safe learning is one of the major obstacles towards real-world agents that can learn during deployment. One way of ensuring that agents respect hard limitations is to explicitly configure boundaries in which they can operate. While this might work in some cases, we do not always have clear a-priori information which states and actions can lead dangerously close to hazardous states. Here, we present an approach where an additional policy can override the main policy and offer a safer alternative action. In our instinct-regulated RL (IR^2L) approach, an "instinctual" network is trained to recognize undesirable situations, while guarding the learning policy against entering them. The instinct network is pre-trained on a single task where it is safe to make mistakes, and transferred to environments in which learning a new task safely is critical. We demonstrate IR^2L in the OpenAI Safety gym domain, in which it receives a significantly lower number of safety violations during training than a baseline RL approach while reaching similar task performance.