mlsanityData to 7 Oct 2026

Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models

arXiv 1507.00814 ↗First posted 3 Jul 2015cs.AI cs.LG stat.MLcited by 154

Not on the map. Why?

Abstract (arXiv)

Achieving efficient and scalable exploration in complex domains poses a major challenge in reinforcement learning. While Bayesian and PAC-MDP approaches to the exploration problem offer strong formal guarantees, they are often impractical in higher dimensions due to their reliance on enumerating the state-action space. Hence, exploration in complex domains is often performed with simple epsilon-greedy methods. In this paper, we consider the challenging Atari games domain, which requires processing raw pixel inputs and delayed rewards. We evaluate several more sophisticated exploration strategies, including Thompson sampling and Boltzman exploration, and propose a new exploration method based on assigning exploration bonuses from a concurrently learned model of the system dynamics. By parameterizing our learned model with a neural network, we are able to develop a scalable and efficient approach to exploration bonuses that can be applied to tasks with complex, high-dimensional state spaces. In the Atari domain, our method provides the most consistent improvement across a range of games that pose a major challenge for prior methods. In addition to raw game-scores, we also develop an AUC-100 metric for the Atari Learning domain to evaluate the impact of exploration on this benchmark.

Key facts. Cited by 154 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Jul 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 33; 2022: 34; 2023: 33; 2024: 20; 2025: 24; 2026: 9.

154 papers.

Sort:

By area:

  1. 2018Self-Imitation Learningcited by 135
  2. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 118
  3. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 116
  4. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
  5. 2021A Survey of Exploration Methods in Reinforcement Learningcited by 68
  6. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 60
  7. 2022Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexitycited by 59
  8. 2021Cooperative Exploration for Multi-Agent Deep Reinforcement Learningcited by 57
  9. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  10. 2022Exploration via Elliptical Episodic Bonusescited by 51
  11. 2021Towards Robust Bisimulation Metric Learningcited by 49
  12. 2021Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Explorationcited by 49
38 more
  1. 2022Improving Intrinsic Exploration with Language Abstractionscited by 49
  2. 2021Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planningcited by 43
  3. 2022Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Modelscited by 42
  4. 2025Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoidscited by 40
  5. 2021Interesting Object, Curious Agent: Learning Task-Agnostic Explorationcited by 38
  6. 2021Mismatched No More: Joint Model-Policy Optimization for Model-Based RLcited by 35
  7. 2023Optimistic Active Exploration of Dynamical Systemscited by 32
  8. 2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 32
  9. 2024MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximizationcited by 30
  10. 2021Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environmentscited by 28
  11. 2023Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learningcited by 28
  12. 2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 25
  13. 2021MURAL: Meta-Learning Uncertainty-Aware Rewards for Outcome-Driven Reinforcement Learningcited by 24
  14. 2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
  15. 2021What Can I Do Here? Learning New Skills by Imagining Visual Affordancescited by 22
  16. 2023Accelerating Reinforcement Learning with Value-Conditional State Entropy Explorationcited by 22
  17. 2023Accelerating Exploration with Unlabeled Prior Datacited by 21
  18. 2021UAV-assisted Online Machine Learning over Multi-Tiered Networks: A Hierarchical Nested Personalized Federated Learning Approachcited by 20
  19. 2023Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Explorationcited by 20
  20. 2024A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoalscited by 20
  21. 2025Temporal Difference Flowscited by 20
  22. 2025Fast Adaptation with Behavioral Foundation Modelscited by 20
  23. 2023A Study of Global and Episodic Bonuses for Exploration in Contextual MDPscited by 19
  24. 2022R'enyi State Entropy for Exploration Acceleration in Reinforcement Learningcited by 18
  25. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  26. 2021Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitationcited by 17
  27. 2022Understanding the Complexity Gains of Single-Task RL with a Curriculumcited by 16
  28. 2023Curious Replay for Model-based Adaptationcited by 16
  29. 2021Decoupled Exploration and Exploitation Policies for Sample-Efficient Reinforcement Learningcited by 14
  30. 2024Can Learned Optimization Make Reinforcement Learning Less Difficult?cited by 14
  31. 2025Curiosity-Driven Reinforcement Learning from Human Feedbackcited by 14
  32. 2021Near-Optimal Reward-Free Exploration for Linear Mixture MDPs with Plug-in Solvercited by 13
  33. 2024Leveraging Skills from Unlabeled Prior Data for Efficient Online Explorationcited by 13
  34. 2026World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetrycited by 13
  35. 2021Exploration and preference satisfaction trade-off in reward-free learningcited by 12
  36. 2022Reinforcement Learning with Automated Auxiliary Loss Searchcited by 12
  37. 2021Information is Power: Intrinsic Control via Information Capturecited by 11
  38. 2022ELIGN: Expectation Alignment as a Multi-Agent Intrinsic Rewardcited by 11

50 most cited of 154 All 154 (JSON)