mlsanityData to 14 Oct 2026

Unlocking the Power of Representations in Long-term Novelty-based Exploration

arXiv 2305.01521 ↗First posted 2 May 2023cs.LG stat.MLcited by 9

Not on the map. Why?

Abstract (arXiv)

We introduce Robust Exploration via Clustering-based Online Density Estimation (RECODE), a non-parametric method for novelty-based exploration that estimates visitation counts for clusters of states based on their similarity in a chosen embedding space. By adapting classical clustering to the nonstationary setting of Deep RL, RECODE can efficiently track state visitation counts over thousands of episodes. We further propose a novel generalization of the inverse dynamics loss, which leverages masked transformer architectures for multi-step prediction; which in conjunction with RECODE achieves a new state-of-the-art in a suite of challenging 3D-exploration tasks in DM-Hard-8. RECODE also sets new state-of-the-art in hard exploration Atari games, and is the first agent to reach the end screen in "Pitfall!".

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 32 of its 64 references that are in the map. First posted on arXiv 2 May 2023.

Builds on

32 of 64 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  3. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  4. 2015Prioritized Experience Replaycited by 1,099
  5. 2018Exploration by Random Network Distillationcited by 993
  6. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  7. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  8. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  9. 2017Minimax Regret Bounds for Reinforcement Learningcited by 503
  10. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  11. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 303
  12. 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
20 more
  1. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  2. 2017Count-Based Exploration with Neural Density Modelscited by 257
  3. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  4. 2020Never Give Up: Learning Directed Exploration Strategiescited by 212
  5. 2019Efficient Exploration via State Marginal Matchingcited by 188
  6. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  7. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 174
  8. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 174
  9. 2017Neural Episodic Controlcited by 151
  10. 2018Episodic Curiosity through Reachabilitycited by 109
  11. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 108
  12. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 102
  13. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 73
  14. 2019Making Efficient Use of Demonstrations to Solve Hard Exploration Problemscited by 45
  15. 2021Geometric Entropic Explorationcited by 32
  16. 2018Back to Basics: Benchmarking Canonical Evolution Strategies for Playing Ataricited by 28
  17. 2022Human-level Atari 200x fastercited by 28
  18. 2020Novelty Search in Representational Space for Sample Efficient Explorationcited by 26
  19. 2020Kernel-Based Reinforcement Learning: A Finite-Time Analysiscited by 23
  20. 2014Practical Kernel-Based Reinforcement Learningcited by 14

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 5; 2025: 3.

9 papers.

  1. 2023Generative Flow Networks as Entropy-Regularized RLcited by 64
  2. 2024RLeXplore: Accelerating Research in Intrinsically-Motivated Reinforcement Learningcited by 10
  3. 2024Can foundation models actively gather information in interactive environments to test hypotheses?cited by 8
  4. 2024Dr. Strategy: Model-Based Generalist Agents with Strategic Dreamingcited by 6
  5. 2024An Empirical Study on the Power of Future Prediction in Partially Observable Environmentscited by 4
  6. 2024Just Cluster It: An Approach for Exploration in High-Dimensions using Clustering and Pre-Trained Representationscited by 3
  7. 2025Beyond Noisy-TVs: Noise-Robust Exploration Via Learning Progress Monitoringcited by 2
  8. 2025The Learning Approach to Gamescited by 0
  9. 2025Meta-Learning to Explore via Memory Density Feedbackcited by 0