mlsanityData to 7 Oct 2026

#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning

arXiv 1611.04717 ↗First posted 15 Nov 2016cs.AI cs.LGcited by 307

Not on the map. Why?

Cited by 20 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Count-based exploration algorithms are known to perform near-optimally when used in conjunction with tabular reinforcement learning (RL) methods for solving small discrete Markov decision processes (MDPs). It is generally thought that count-based methods cannot be applied in high-dimensional state spaces, since most states will only occur once. Recent deep RL exploration strategies are able to deal with high-dimensional continuous state spaces through complex heuristics, often relying on optimism in the face of uncertainty or intrinsic motivation. In this work, we describe a surprising finding: a simple generalization of the classic count-based approach can reach near state-of-the-art performance on various high-dimensional and/or continuous deep RL benchmarks. States are mapped to hash codes, which allows to count their occurrences with a hash table. These counts are then used to compute a reward bonus according to the classic count-based exploration theory. We find that simple hash functions can achieve surprisingly good results on many challenging tasks. Furthermore, we show that a domain-dependent learned hash code may further improve these results. Detailed analysis reveals important aspects of a good hash function: 1) having appropriate granularity and 2) encoding information relevant to solving the MDP. This exploration strategy achieves near state-of-the-art performance on both continuous control tasks and Atari 2600 games, hence providing a simple yet powerful baseline for solving MDPs that require considerable exploration.

Key facts. Cited by 307 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 2; 2019: 2; 2020: 0; 2021: 64; 2022: 59; 2023: 58; 2024: 45; 2025: 47; 2026: 30.

307 papers.

Sort:

By area:

  1. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  2. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  3. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  4. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 158
  5. 2018Self-Imitation Learningcited by 135
  6. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 116
  7. 2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 108
  8. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
  9. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  10. 2021DEUP: Direct Epistemic Uncertainty Predictioncited by 102
  11. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  12. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 71
38 more
  1. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 60
  2. 2021Reinforcement Learning, Bit by Bitcited by 59
  3. 2022Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexitycited by 59
  4. 2021Cooperative Exploration for Multi-Agent Deep Reinforcement Learningcited by 57
  5. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  6. 2022Exploration via Elliptical Episodic Bonusescited by 51
  7. 2021Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Explorationcited by 49
  8. 2022Improving Intrinsic Exploration with Language Abstractionscited by 49
  9. 2021Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learningcited by 46
  10. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 45
  11. 2021Offline Reinforcement Learning as Anti-Explorationcited by 44
  12. 2021Curious Representation Learning for Embodied Intelligencecited by 43
  13. 2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
  14. 2025Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoidscited by 40
  15. 2021Interesting Object, Curious Agent: Learning Task-Agnostic Explorationcited by 38
  16. 2021Reinforcement Learning for Adaptive Mesh Refinementcited by 37
  17. 2021On Bonus-Based Exploration Methods in the Arcade Learning Environmentcited by 37
  18. 2021Hierarchical Skills for Efficient Explorationcited by 37
  19. 2022A Review of Uncertainty for Deep Reinforcement Learningcited by 36
  20. 2021Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomizationcited by 35
  21. 2024Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applicationscited by 34
  22. 2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 32
  23. 2023End-to-End Meta-Bayesian Optimisation with Transformer Neural Processescited by 31
  24. 2023Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlocited by 29
  25. 2021Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environmentscited by 28
  26. 2021Provable RL with Exogenous Distractors via Multistep Inverse Dynamicscited by 28
  27. 2023Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learningcited by 28
  28. 2025CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Modelscited by 28
  29. 2021Principled Exploration via Optimistic Bootstrapping and Backward Inductioncited by 26
  30. 2021Decoupled Reinforcement Learning to Stabilise Intrinsically-Motivated Explorationcited by 25
  31. 2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 25
  32. 2021MURAL: Meta-Learning Uncertainty-Aware Rewards for Outcome-Driven Reinforcement Learningcited by 24
  33. 2023The SocialAI School: Insights from Developmental Psychology Towards Artificial Socio-Cultural Agentscited by 24
  34. 2025Reward Models in Deep Reinforcement Learning: A Surveycited by 24
  35. 2021Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Surveycited by 23
  36. 2023ALAN: Autonomously Exploring Robotic Agents in the Real Worldcited by 23
  37. 2021Curiosity-Driven Exploration via Latent Bayesian Surprisecited by 22
  38. 2021What Can I Do Here? Learning New Skills by Imagining Visual Affordancescited by 22

50 most cited of 307 All 307 (JSON)