mlsanityData to 7 Oct 2026

First return, then explore

arXiv 2004.12919 ↗First posted 27 Apr 2020cs.AIcited by 256

Not on the map. Why?

Abstract (arXiv)

The promise of reinforcement learning is to solve complex sequential decision problems autonomously by specifying a high-level reward function only. However, reinforcement learning algorithms struggle when, as is often the case, simple and intuitive rewards provide sparse and deceptive feedback. Avoiding these pitfalls requires thoroughly exploring the environment, but creating algorithms that can do so remains one of the central challenges of the field. We hypothesise that the main impediment to effective exploration originates from algorithms forgetting how to reach previously visited states ("detachment") and from failing to first return to a state before exploring from it ("derailment"). We introduce Go-Explore, a family of algorithms that addresses these two challenges directly through the simple principles of explicitly remembering promising states and first returning to such states before intentionally exploring. Go-Explore solves all heretofore unsolved Atari games and surpasses the state of the art on all hard-exploration games, with orders of magnitude improvements on the grand challenges Montezuma's Revenge and Pitfall. We also demonstrate the practical potential of Go-Explore on a sparse-reward pick-and-place robotics task. Additionally, we show that adding a goal-conditioned policy can further improve Go-Explore's exploration efficiency and enable it to handle stochasticity throughout training. The substantial performance gains from Go-Explore suggest that the simple principles of remembering states, returning to them, and exploring from them are a powerful and general approach to exploration, an insight that may prove critical to the creation of truly intelligent learning agents.

Key facts. Cited by 256 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 31; 2022: 64; 2023: 54; 2024: 35; 2025: 41; 2026: 31.

256 papers.

Sort:

By area:

  1. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  2. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  3. 2022Jump-Start Reinforcement Learningcited by 102
  4. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83
  5. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  6. 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 80
  7. 2021Reinforcement learning for optimization of variational quantum circuit architecturescited by 76
  8. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 71
  9. 2021Differentiable Quality Diversitycited by 71
  10. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 71
  11. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  12. 2025Embodied AI Agents: Modeling the Worldcited by 57
38 more
  1. 2022Semantic Exploration from Language Abstractions and Pretrained Representationscited by 51
  2. 2025First Return, Entropy-Eliciting Explorecited by 45
  3. 2022Atari-5: Distilling the Arcade Learning Environment down to Five Gamescited by 44
  4. 2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
  5. 2023Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Predictioncited by 39
  6. 2022ActMAD: Activation Matching to Align Distributions for Test-Time-Trainingcited by 38
  7. 2021Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraftcited by 34
  8. 2024Intelligent Go-Explore: Standing on the Shoulders of Giant Foundation Modelscited by 32
  9. 2025Go-Browse: Training Web Agents with Structured Explorationcited by 32
  10. 2021Unsupervised Behaviour Discovery with Quality-Diversity Optimisationcited by 30
  11. 2023OCAtari: Object-Centric Atari 2600 Reinforcement Learning Environmentscited by 30
  12. 2023Augmenting Autotelic Agents with Large Language Modelscited by 29
  13. 2022Human-level Atari 200x fastercited by 28
  14. 2023Deep Laplacian-based Options for Temporally-Extended Explorationcited by 28
  15. 2023Planning Goals for Explorationcited by 28
  16. 2023Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learningcited by 28
  17. 2022DEP-RL: Embodied Exploration for Reinforcement Learning in Overactuated and Musculoskeletal Systemscited by 27
  18. 2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 25
  19. 2025PoE-World: Compositional World Modeling with Products of Programmatic Expertscited by 25
  20. 2022Redeeming Intrinsic Rewards via Constrained Optimizationcited by 24
  21. 2023Evolutionary Reinforcement Learning: A Surveycited by 24
  22. 2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
  23. 2022Multi-Stage Episodic Control for Strategic Exploration in Text Gamescited by 22
  24. 2022PrefRec: Recommender Systems with Human Preferences for Reinforcing Long-term User Engagementcited by 21
  25. 2023Latent Exploration for Reinforcement Learningcited by 21
  26. 2021Dynamics-Aware Quality-Diversity for Efficient Learning of Skill Repertoirescited by 20
  27. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  28. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensemblescited by 20
  29. 2022Reinforcement Learning for Branch-and-Bound Optimisation using Retrospective Trajectoriescited by 19
  30. 2022Parametrically Retargetable Decision-Makers Tend To Seek Powercited by 19
  31. 2025Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesiscited by 19
  32. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  33. 2023TiZero: Mastering Multi-Agent Football with Curriculum Learning and Self-Playcited by 18
  34. 2023Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Groundingcited by 18
  35. 2021Hyperparameter Selection for Imitation Learningcited by 17
  36. 2024Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learningcited by 17
  37. 2024TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representationscited by 17
  38. 2022Phasic Self-Imitative Reduction for Sparse-Reward Goal-Conditioned Reinforcement Learningcited by 16

50 most cited of 256 All 256 (JSON)