# Episodic Curiosity through Reachability

arXiv: https://arxiv.org/abs/1810.02274 · HTML: https://mlsanity.com/paper/1810.02274/

## Key facts

- Cited by 109 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Oct 2018.

## Built on by (most cited first, 30 of 109)

- [A Survey of Embodied AI: From Simulators to Research Tasks](https://mlsanity.com/paper/2103.04918/) (2021)
- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)
- [Rapid Exploration for Open-World Navigation with Latent Goal Models](https://mlsanity.com/paper/2104.05859/) (2021)
- [Navigating to Objects in the Real World](https://mlsanity.com/paper/2212.00922/) (2022)
- [Memory-Augmented Reinforcement Learning for Image-Goal Navigation](https://mlsanity.com/paper/2101.05181/) (2021)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [RobustNav: Towards Benchmarking Robustness in Embodied Navigation](https://mlsanity.com/paper/2106.04531/) (2021)
- [Semantic Exploration from Language Abstractions and Pretrained Representations](https://mlsanity.com/paper/2204.05080/) (2022)
- [SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2108.11945/) (2021)
- [Retrospectives on the Embodied AI Workshop](https://mlsanity.com/paper/2210.06849/) (2022)
- [Geometric Entropic Exploration](https://mlsanity.com/paper/2101.02055/) (2021)
- [Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments](https://mlsanity.com/paper/2101.08152/) (2021)
- [Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey](https://mlsanity.com/paper/2110.01411/) (2021)
- [An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey](https://mlsanity.com/paper/2209.08890/) (2022)
- [Principled Exploration via Optimistic Bootstrapping and Backward Induction](https://mlsanity.com/paper/2105.06022/) (2021)
- [Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping](https://mlsanity.com/paper/2301.02099/) (2023)
- [The SocialAI School: Insights from Developmental Psychology Towards Artificial Socio-Cultural Agents](https://mlsanity.com/paper/2307.07871/) (2023)
- [Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2111.09858/) (2021)
- [Open Long-Tailed Recognition in a Dynamic World](https://mlsanity.com/paper/2208.08349/) (2022)
- [When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learning](https://mlsanity.com/paper/2210.10765/) (2022)
- [Megaverse: Simulating Embodied Agents at One Million Experiences per Second](https://mlsanity.com/paper/2107.08170/) (2021)
- [ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles](https://mlsanity.com/paper/2306.06871/) (2023)
- [Learning Efficient Multi-Agent Cooperative Visual Exploration](https://mlsanity.com/paper/2110.05734/) (2021)
- [There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learning](https://mlsanity.com/paper/2106.04480/) (2021)
- [R'enyi State Entropy for Exploration Acceleration in Reinforcement Learning](https://mlsanity.com/paper/2203.04297/) (2022)
- [Unsupervised Representation Learning in Deep Reinforcement Learning: A Review](https://mlsanity.com/paper/2208.14226/) (2022)
- [Asynchronous Multi-Agent Reinforcement Learning for Efficient Real-Time Multi-Robot Cooperative Exploration](https://mlsanity.com/paper/2301.03398/) (2023)
- [Shaping embodied agent behavior with activity-context priors from egocentric video](https://mlsanity.com/paper/2110.07692/) (2021)
- [Navigating to Objects in Unseen Environments by Distance Prediction](https://mlsanity.com/paper/2202.03735/) (2022)
- [Posterior Sampling for Deep Reinforcement Learning](https://mlsanity.com/paper/2305.00477/) (2023)

## Abstract (arXiv)

Rewards are sparse in the real world and most of today's reinforcement learning algorithms struggle with such sparsity. One solution to this problem is to allow the agent to create rewards for itself - thus making rewards dense and more suitable for learning. In particular, inspired by curious behaviour in animals, observing something novel could be rewarded with a bonus. Such bonus is summed up with the real task reward - making it possible for RL algorithms to learn from the combined reward. We propose a new curiosity method which uses episodic memory to form the novelty bonus. To determine the bonus, the current observation is compared with the observations in memory. Crucially, the comparison is done based on how many environment steps it takes to reach the current observation from those in memory - which incorporates rich information about environment dynamics. This allows us to overcome the known "couch-potato" issues of prior work - when the agent finds a way to instantly gratify itself by exploiting actions which lead to hardly predictable consequences. We test our approach in visually rich 3D environments in ViZDoom, DMLab and MuJoCo. In navigational tasks from ViZDoom and DMLab, our agent outperforms the state-of-the-art curiosity method ICM. In MuJoCo, an ant equipped with our curiosity module learns locomotion out of the first-person-view curiosity only.