Data to 7 Oct 2026

Count-Based Exploration with Neural Density Models

arXiv 1703.01310 ↗First posted 3 Mar 2017cs.AIcited by 255

Not on the map. Why?

Abstract (arXiv)

Bellemare et al. (2016) introduced the notion of a pseudo-count, derived from a density model, to generalize count-based exploration to non-tabular reinforcement learning. This pseudo-count was used to generate an exploration bonus for a DQN agent and combined with a mixed Monte Carlo update was sufficient to achieve state of the art on the Atari 2600 game Montezuma's Revenge. We consider two questions left open by their work: First, how important is the quality of the density model for exploration? Second, what role does the Monte Carlo update play in exploration? We answer the first question by demonstrating the use of PixelCNN, an advanced neural density model for images, to supply a pseudo-count. In particular, we examine the intrinsic difficulties in adapting Bellemare et al.'s approach when assumptions about the model are violated. The result is a more practical and general algorithm requiring no special apparatus. We combine PixelCNN pseudo-counts with different agent architectures to dramatically improve the state of the art on several hard Atari games. One surprising finding is that the mixed Monte Carlo update is a powerful facilitator of exploration in the sparsest of settings, including Montezuma's Revenge.

Key facts. Cited by 255 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Mar 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 2; 2019: 0; 2020: 0; 2021: 54; 2022: 44; 2023: 57; 2024: 46; 2025: 32; 2026: 20.

255 papers.

Sort:

By area:

2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
2021Reinforcement Learning with Prototypical Representationscited by 179
2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
2022Exploration in Deep Reinforcement Learning: A Surveycited by 158
2018Self-Imitation Learningcited by 135
2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 74

38 more

2021A Survey of Exploration Methods in Reinforcement Learningcited by 68
2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 60
2023Generative Flow Networks as Entropy-Regularized RLcited by 60
2021Cooperative Exploration for Multi-Agent Deep Reinforcement Learningcited by 57
2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
2022Exploration via Elliptical Episodic Bonusescited by 51
2021Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Explorationcited by 49
2022Improving Intrinsic Exploration with Language Abstractionscited by 49
2021Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learningcited by 46
2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 45
2024Efficient Exploration for LLMscited by 44
2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
2025Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoidscited by 40
2021Interesting Object, Curious Agent: Learning Task-Agnostic Explorationcited by 38
2021On Bonus-Based Exploration Methods in the Arcade Learning Environmentcited by 37
2023Neural Categorical Priors for Physics-Based Character Controlcited by 37
2025Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Explorationcited by 33
2021Geometric Entropic Explorationcited by 32
2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 32
2023Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlocited by 29
2021Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environmentscited by 28
2023Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learningcited by 28
2021Continuous Doubly Constrained Batch Reinforcement Learningcited by 27
2021Principled Exploration via Optimistic Bootstrapping and Backward Inductioncited by 26
2021Decoupled Reinforcement Learning to Stabilise Intrinsically-Motivated Explorationcited by 25
2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 25
2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
2025Reward Models in Deep Reinforcement Learning: A Surveycited by 24
2021Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Surveycited by 23
2023ALAN: Autonomously Exploring Robotic Agents in the Real Worldcited by 23
2021Curiosity-Driven Exploration via Latent Bayesian Surprisecited by 22
2022From Dirichlet to Rubin: Optimistic Exploration in RL without Bonusescited by 22
2023Accelerating Reinforcement Learning with Value-Conditional State Entropy Explorationcited by 22
2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generationcited by 22
2021Dynamic Bottleneck for Robust Self-Supervised Explorationcited by 21
2024Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learningcited by 21
2021When should agents explore?cited by 20
2022General Intelligence Requires Rethinking Explorationcited by 20

50 most cited of 255 All 255 (JSON)