# Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear

arXiv: https://arxiv.org/abs/1611.01211 · HTML: https://mlsanity.com/paper/1611.01211/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Nov 2016.

## Built on by (most cited first, 5 of 5)

- [Improving Safety in Deep Reinforcement Learning using Unsupervised Action Planning](https://mlsanity.com/paper/2109.14325/) (2021)
- [Safe Deep Reinforcement Learning for Multi-Agent Systems with Continuous Action Spaces](https://mlsanity.com/paper/2108.03952/) (2021)
- [Knowledge-Guided Exploration in Deep Reinforcement Learning](https://mlsanity.com/paper/2210.15670/) (2022)
- [Zero-Shot Uncertainty-Aware Deployment of Simulation Trained Policies on Real-World Robots](https://mlsanity.com/paper/2112.05299/) (2021)
- [Advances in Electron Microscopy with Deep Learning](https://mlsanity.com/paper/2101.01178/) (2021)

## Abstract (arXiv)

Many practical environments contain catastrophic states that an optimal agent would visit infrequently or never. Even on toy problems, Deep Reinforcement Learning (DRL) agents tend to periodically revisit these states upon forgetting their existence under a new policy. We introduce intrinsic fear (IF), a learned reward shaping that guards DRL agents against periodic catastrophes. IF agents possess a fear model trained to predict the probability of imminent catastrophe. This score is then used to penalize the Q-learning objective. Our theoretical analysis bounds the reduction in average return due to learning on the perturbed objective. We also prove robustness to classification errors. As a bonus, IF models tend to learn faster, owing to reward shaping. Experiments demonstrate that intrinsic-fear DQNs solve otherwise pathological environments and improve on several Atari games.