mlsanityData to 14 Oct 2026

Curiosity-driven Exploration by Self-supervised Prediction

arXiv 1705.05363 ↗First posted 15 May 2017cs.LG cs.AI cs.CVcited by 1,194

Not on the map. Why?

Cited by 282 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In many real-world scenarios, rewards extrinsic to the agent are extremely sparse, or absent altogether. In such cases, curiosity can serve as an intrinsic reward signal to enable the agent to explore its environment and learn skills that might be useful later in its life. We formulate curiosity as the error in an agent's ability to predict the consequence of its own actions in a visual feature space learned by a self-supervised inverse dynamics model. Our formulation scales to high-dimensional continuous state spaces like images, bypasses the difficulties of directly predicting pixels, and, critically, ignores the aspects of the environment that cannot affect the agent. The proposed approach is evaluated in two environments: VizDoom and Super Mario Bros. Three broad settings are investigated: 1) sparse extrinsic reward, where curiosity allows for far fewer interactions with the environment to reach the goal; 2) exploration with no extrinsic reward, where curiosity pushes the agent to explore more efficiently; and 3) generalization to unseen scenarios (e.g. new levels of the same game) where the knowledge gained from earlier experience helps the agent explore new places much faster than starting from scratch. Demo video and code available at https://pathak22.github.io/noreward-rl/

Key facts. Cited by 1,194 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 3; 2019: 0; 2020: 4; 2021: 200; 2022: 181; 2023: 218; 2024: 158; 2025: 213; 2026: 217.

1,194 papers.

Sort:

By area:

  1. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  2. 2021Generalized Out-of-Distribution Detection: A Surveycited by 802
  3. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
  4. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
  5. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  6. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
  7. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 278
  8. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 261
  9. 2023NoMaD: Goal Masked Diffusion Policies for Navigation and Explorationcited by 248
  10. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 226
  11. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
  12. 2024DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Searchcited by 197
38 more
  1. 2022Active Example Selection for In-Context Learningcited by 196
  2. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  3. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 187
  4. 2021Reinforcement Learning with Prototypical Representationscited by 179
  5. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
  6. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 164
  7. 2022Human-to-Robot Imitation in the Wildcited by 161
  8. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 148
  9. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 143
  10. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 139
  11. 2020PLAS: Latent Action Space for Offline Reinforcement Learningcited by 133
  12. 2021Discovering and Achieving Goals via World Modelscited by 131
  13. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 121
  14. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 117
  15. 2021APS: Active Pretraining with Successor Featurescited by 112
  16. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 108
  17. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 108
  18. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  19. 2025Maximizing Confidence Alone Improves Reasoningcited by 100
  20. 2021Celebrating Diversity in Shared Multi-Agent Reinforcement Learningcited by 98
  21. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 96
  22. 2024Foundation Policies with Hilbert Representationscited by 94
  23. 2020State-Only Imitation Learning for Dexterous Manipulationcited by 91
  24. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 91
  25. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
  26. 2024Curiosity-driven Red-teaming for Large Language Modelscited by 88
  27. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 86
  28. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 86
  29. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 83
  30. 2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
  31. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 82
  32. 2021VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objectscited by 80
  33. 2022A Theoretical Study on Solving Continual Learningcited by 80
  34. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 80
  35. 2021Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformerscited by 79
  36. 2025Tree Search for LLM Agent Reinforcement Learningcited by 79
  37. 2021Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Taskscited by 78
  38. 2021Auxiliary Tasks and Exploration Enable ObjectNavcited by 74

50 most cited of 1,194 All 1,194 (JSON)