mlsanityData to 7 Oct 2026

Unifying Count-Based Exploration and Intrinsic Motivation

arXiv 1606.01868 ↗First posted 6 Jun 2016cs.AI cs.LG stat.MLcited by 632

Not on the map. Why?

Cited by 38 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We consider an agent's uncertainty about its environment and the problem of generalizing this uncertainty across observations. Specifically, we focus on the problem of exploration in non-tabular reinforcement learning. Drawing inspiration from the intrinsic motivation literature, we use density models to measure uncertainty, and propose a novel algorithm for deriving a pseudo-count from an arbitrary density model. This technique enables us to generalize count-based exploration algorithms to the non-tabular case. We apply our ideas to Atari 2600 games, providing sensible pseudo-counts from raw pixels. We transform these pseudo-counts into intrinsic rewards and obtain significantly improved exploration in a number of hard games, including the infamously difficult Montezuma's Revenge.

Key facts. Cited by 632 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Jun 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 2; 2019: 1; 2020: 1; 2021: 122; 2022: 116; 2023: 118; 2024: 104; 2025: 86; 2026: 82.

632 papers.

Sort:

By area:

  1. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  2. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  3. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  4. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 267
  5. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  6. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  7. 2024DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Searchcited by 189
  8. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  9. 2021Reinforcement Learning with Prototypical Representationscited by 179
  10. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  11. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 158
  12. 2018Self-Imitation Learningcited by 135
38 more
  1. 2021Discovering and Achieving Goals via World Modelscited by 131
  2. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 118
  3. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 116
  4. 2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 108
  5. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
  6. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  7. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  8. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 96
  9. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  10. 2024Curiosity-driven Red-teaming for Large Language Modelscited by 83
  11. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  12. 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 80
  13. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 79
  14. 2025Outcome-based Exploration for LLM Reasoningcited by 79
  15. 2021Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Taskscited by 75
  16. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 71
  17. 2021A Survey of Exploration Methods in Reinforcement Learningcited by 68
  18. 2021Hit and Lead Discovery with Explorative RL and Fragment-based Molecule Generationcited by 68
  19. 2023OMNI: Open-endedness via Models of human Notions of Interestingnesscited by 66
  20. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 60
  21. 2023Generative Flow Networks as Entropy-Regularized RLcited by 60
  22. 2022Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexitycited by 59
  23. 2021Cooperative Exploration for Multi-Agent Deep Reinforcement Learningcited by 57
  24. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  25. 2022ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hintscited by 55
  26. 2022Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approachcited by 51
  27. 2022Semantic Exploration from Language Abstractions and Pretrained Representationscited by 51
  28. 2022Exploration via Elliptical Episodic Bonusescited by 51
  29. 2021Towards Robust Bisimulation Metric Learningcited by 49
  30. 2021Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Explorationcited by 49
  31. 2022Improving Intrinsic Exploration with Language Abstractionscited by 49
  32. 2021Adversarially Guided Actor-Criticcited by 47
  33. 2021Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learningcited by 46
  34. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 45
  35. 2024Learning Formal Mathematics From Intrinsic Motivationcited by 45
  36. 2021Offline Reinforcement Learning as Anti-Explorationcited by 44
  37. 2022Atari-5: Distilling the Arcade Learning Environment down to Five Gamescited by 44
  38. 2024Efficient Exploration for LLMscited by 44

50 most cited of 632 All 632 (JSON)