mlsanityData to 14 Oct 2026

Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learning

arXiv 2306.03186 ↗First posted 5 Jun 2023cs.LG cs.AIcited by 28

Not on the map. Why?

Abstract (arXiv)

We propose a new method for count-based exploration in high-dimensional state spaces. Unlike previous work which relies on density models, we show that counts can be derived by averaging samples from the Rademacher distribution (or coin flips). This insight is used to set up a simple supervised learning objective which, when optimized, yields a state's visitation count. We show that our method is significantly more effective at deducing ground-truth visitation counts than previous work; when used as an exploration bonus for a model-free reinforcement learning algorithm, it outperforms existing approaches on most of 9 challenging exploration tasks, including the Atari game Montezuma's Revenge.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 36 of its 56 references that are in the map. First posted on arXiv 5 Jun 2023.

Builds on

36 of 56 references.

References · 36

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  5. 2018Exploration by Random Network Distillationcited by 993
  6. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  7. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  8. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  9. 2016Deep Exploration via Bootstrapped DQNcited by 527
  10. 2017Minimax Regret Bounds for Reinforcement Learningcited by 503
  11. 2018Is Q-learning Provably Efficient?cited by 487
  12. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
24 more
  1. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 303
  2. 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
  3. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  4. 2020First return, then explorecited by 266
  5. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
  6. 2017Count-Based Exploration with Neural Density Modelscited by 257
  7. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 219
  8. 2020Never Give Up: Learning Directed Exploration Strategiescited by 212
  9. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 174
  10. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  11. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  12. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
  13. 2018Count-Based Exploration with the Successor Representationcited by 112
  14. 2017A Deeper Look at Experience Replaycited by 105
  15. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 102
  16. 2017The Uncertainty Bellman Equation and Explorationcited by 82
  17. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 73
  18. 2021Learning Markov State Abstractions for Deep Reinforcement Learningcited by 44
  19. 2021On Bonus-Based Exploration Methods in the Arcade Learning Environmentcited by 37
  20. 2022Human-level Atari 200x fastercited by 28
  21. 2020Optimistic Exploration even with a Pessimistic Initialisationcited by 25
  22. 2022The Alberta Plan for AI Researchcited by 18
  23. 2020Latent World Models For Intrinsically Motivated Explorationcited by 15
  24. 2014Domain-Independent Optimistic Initialization for Reinforcement Learningcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 10; 2025: 10; 2026: 6.

28 papers.

Sort:

By area:

  1. 2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 33
  2. 2024Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learningcited by 22
  3. 2025Episodic Novelty Through Temporal Distancecited by 16
  4. 2024Leveraging Skills from Unlabeled Prior Data for Efficient Online Explorationcited by 13
  5. 2025EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Controlcited by 13
  6. 2024RLeXplore: Accelerating Research in Intrinsically-Motivated Reinforcement Learningcited by 10
  7. 2024An Optimal Tightness Bound for the Simulation Lemmacited by 8
  8. 2023Hierarchical Empowerment: Towards Tractable Empowerment-Based Skill Learningcited by 7
  9. 2023Improving Intrinsic Exploration by Creating Stationary Objectivescited by 7
  10. 2024Beyond Optimism: Exploration With Partially Observable Rewardscited by 6
  11. 2025Uncertainty Prioritized Experience Replaycited by 6
  12. 2025Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigationcited by 5
16 more
  1. 2026Emergence of Exploration in Policy Gradient Reinforcement Learning via Retryingcited by 5
  2. 2024Surprise-Adaptive Intrinsic Motivation for Unsupervised Reinforcement Learningcited by 4
  3. 2024BAMDP Shaping: a Unified Framework for Intrinsic Motivation and Reward Shapingcited by 4
  4. 2024Novelty-Guided Data Reuse for Efficient and Diversified Multi-Agent Reinforcement Learningcited by 4
  5. 2025β-DQN: Improving Deep Q-Learning By Evolving the Behaviorcited by 3
  6. 2024ETGL-DDPG: A Deep Deterministic Policy Gradient Algorithm for Sparse Reward Continuous Controlcited by 2
  7. 2025Finite-time Convergence Analysis of Actor-Critic with Evolving Rewardcited by 2
  8. 2025Exploration by Random Distribution Distillationcited by 1
  9. 2025Exploration by Random Reward Perturbationcited by 1
  10. 2026Maximum Entropy Exploration Without the Rolloutscited by 1
  11. 2026VLAConf: Calibrated Task-Success Confidence for Vision-Language-Action Modelscited by 1
  12. 2026Retry Policy Gradients in Continuous Action Spacescited by 1
  13. 2026Going Beyond State-Reaching: Learning Abstractions for Intrinsically Motivated Option Discoverycited by 1
  14. 2025Exploration Behavior of Untrained Policiescited by 0
  15. 2025Fill in the Blanks: Accelerating Q-Learning with a Handful of Demonstrations in Sparse Reward Settingscited by 0
  16. 2026CIG: Exploration via Conditional Information Gaincited by 0