# Large-Scale Study of Curiosity-Driven Learning

arXiv: https://arxiv.org/abs/1808.04355 · HTML: https://mlsanity.com/paper/1808.04355/

## Key facts

- Cited by 280 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Aug 2018.

## Built on by (most cited first, 30 of 280)

- [A Survey of Embodied AI: From Simulators to Research Tasks](https://mlsanity.com/paper/2103.04918/) (2021)
- [Benchmarking the Spectrum of Agent Capabilities](https://mlsanity.com/paper/2109.06780/) (2021)
- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)
- [URLB: Unsupervised Reinforcement Learning Benchmark](https://mlsanity.com/paper/2110.15191/) (2021)
- [Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scale](https://mlsanity.com/paper/2204.03514/) (2022)
- [APS: Active Pretraining with Successor Features](https://mlsanity.com/paper/2108.13956/) (2021)
- [Pretraining Representations for Data-Efficient Reinforcement Learning](https://mlsanity.com/paper/2106.04799/) (2021)
- [Motif: Intrinsic Motivation from Artificial Intelligence Feedback](https://mlsanity.com/paper/2310.00166/) (2023)
- [CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization](https://mlsanity.com/paper/2310.10134/) (2023)
- [A Survey of Exploration Methods in Reinforcement Learning](https://mlsanity.com/paper/2109.00157/) (2021)
- [BYOL-Explore: Exploration by Bootstrapped Prediction](https://mlsanity.com/paper/2206.08332/) (2022)
- [CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery](https://mlsanity.com/paper/2202.00161/) (2022)
- [Learning Auxiliary Monocular Contexts Helps Monocular 3D Object Detection](https://mlsanity.com/paper/2112.04628/) (2021)
- [Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft](https://mlsanity.com/paper/2312.09238/) (2023)
- [Reinforcement Learning, Bit by Bit](https://mlsanity.com/paper/2103.04047/) (2021)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [Continual Learning for Real-World Autonomous Systems: Algorithms, Challenges and Frameworks](https://mlsanity.com/paper/2105.12374/) (2021)
- [Semantic Exploration from Language Abstractions and Pretrained Representations](https://mlsanity.com/paper/2204.05080/) (2022)
- [Exploration via Elliptical Episodic Bonuses](https://mlsanity.com/paper/2210.05805/) (2022)
- [Towards Robust Bisimulation Metric Learning](https://mlsanity.com/paper/2110.14096/) (2021)
- [Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Exploration](https://mlsanity.com/paper/2111.11032/) (2021)
- [RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation](https://mlsanity.com/paper/2402.15487/) (2024)
- [Curious Representation Learning for Embodied Intelligence](https://mlsanity.com/paper/2105.01060/) (2021)
- [Offline Reinforcement Learning as Anti-Exploration](https://mlsanity.com/paper/2106.06431/) (2021)
- [ELLA: Exploration through Learned Language Abstraction](https://mlsanity.com/paper/2103.05825/) (2021)
- [MADE: Exploration via Maximizing Deviation from Explored Regions](https://mlsanity.com/paper/2106.10268/) (2021)
- [On Bonus-Based Exploration Methods in the Arcade Learning Environment](https://mlsanity.com/paper/2109.11052/) (2021)
- [Intrinsic Reward Driven Imitation Learning via Generative Model](https://mlsanity.com/paper/2006.15061/) (2020)
- [Exploration and Anti-Exploration with Distributional Random Network Distillation](https://mlsanity.com/paper/2401.09750/) (2024)
- [Unsupervised Learning of Visual 3D Keypoints for Control](https://mlsanity.com/paper/2106.07643/) (2021)

## Abstract (arXiv)

Reinforcement learning algorithms rely on carefully engineering environment rewards that are extrinsic to the agent. However, annotating each environment with hand-designed, dense rewards is not scalable, motivating the need for developing reward functions that are intrinsic to the agent. Curiosity is a type of intrinsic reward function which uses prediction error as reward signal. In this paper: (a) We perform the first large-scale study of purely curiosity-driven learning, i.e. without any extrinsic rewards, across 54 standard benchmark environments, including the Atari game suite. Our results show surprisingly good performance, and a high degree of alignment between the intrinsic curiosity objective and the hand-designed extrinsic rewards of many game environments. (b) We investigate the effect of using different feature spaces for computing prediction error and show that random features are sufficient for many popular RL game benchmarks, but learned features appear to generalize better (e.g. to novel game levels in Super Mario Bros.). (c) We demonstrate limitations of the prediction-based rewards in stochastic setups. Game-play videos and code are at https://pathak22.github.io/large-scale-curiosity/