mlsanityData to 14 Oct 2026

Go-Explore: a New Approach for Hard-Exploration Problems

arXiv 1901.10995 ↗First posted 30 Jan 2019cs.LG cs.AI stat.MLcited by 212

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

A grand challenge in reinforcement learning is intelligent exploration, especially when rewards are sparse or deceptive. Two Atari games serve as benchmarks for such hard-exploration domains: Montezuma's Revenge and Pitfall. On both games, current RL algorithms perform poorly, even those with intrinsic motivation, which is the dominant method to improve performance on hard-exploration domains. To address this shortfall, we introduce a new algorithm called Go-Explore. It exploits the following principles: (1) remember previously visited states, (2) first return to a promising state (without exploration), then explore from it, and (3) solve simulated environments through any available means (including by introducing determinism), then robustify via imitation learning. The combined effect of these principles is a dramatic performance improvement on hard-exploration problems. On Montezuma's Revenge, Go-Explore scores a mean of over 43k points, almost 4 times the previous state of the art. Go-Explore can also harness human-provided domain knowledge and, when augmented with it, scores a mean of over 650k points on Montezuma's Revenge. Its max performance of nearly 18 million surpasses the human world record, meeting even the strictest definition of "superhuman" performance. On Pitfall, Go-Explore with domain knowledge is the first algorithm to score above zero. Its mean score of almost 60k points exceeds expert human performance. Because Go-Explore produces high-performing demonstrations automatically and cheaply, it also outperforms imitation learning work where humans provide solution demonstrations. Go-Explore opens up many new research directions into improving it and weaving its insights into current RL algorithms. It may also enable progress on previously unsolvable hard-exploration problems in many domains, especially those that harness a simulator during training (e.g. robotics).

Key facts. Cited by 212 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. First posted on arXiv 30 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 42; 2022: 33; 2023: 37; 2024: 28; 2025: 40; 2026: 32.

212 papers.

Sort:

By area:

  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  3. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  4. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  5. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  6. 2023Cognitive Architectures for Language Agentscited by 429
  7. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 356
  8. 2021Character Controllers Using Motion VAEscited by 184
  9. 2021Discovering and Achieving Goals via World Modelscited by 131
  10. 2022Jump-Start Reinforcement Learningcited by 112
  11. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 97
  12. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 96
38 more
  1. 2024Curiosity-driven Red-teaming for Large Language Modelscited by 88
  2. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 73
  3. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 72
  4. 2023OMNI: Open-endedness via Models of human Notions of Interestingnesscited by 70
  5. 2021Hit and Lead Discovery with Explorative RL and Fragment-based Molecule Generationcited by 68
  6. 2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrationscited by 62
  7. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 58
  8. 2025Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Searchcited by 53
  9. 2022Exploration via Elliptical Episodic Bonusescited by 51
  10. 2025Self-Questioning Language Modelscited by 50
  11. 2023Learning to Generate Better Than Your LLMcited by 49
  12. 2021Adversarially Guided Actor-Criticcited by 47
  13. 2023Towards A Unified Agent with Foundation Modelscited by 47
  14. 2025First Return, Entropy-Eliciting Explorecited by 45
  15. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 42
  16. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 41
  17. 2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
  18. 2021On Bonus-Based Exploration Methods in the Arcade Learning Environmentcited by 37
  19. 2023Skill Reinforcement Learning and Planning for Open-World Long-Horizon Taskscited by 36
  20. 2021Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraftcited by 35
  21. 2023Fast Rates for Maximum Entropy Explorationcited by 33
  22. 2024Intelligent Go-Explore: Standing on the Shoulders of Giant Foundation Modelscited by 33
  23. 2025Go-Browse: Training Web Agents with Structured Explorationcited by 33
  24. 2023OCAtari: Object-Centric Atari 2600 Reinforcement Learning Environmentscited by 30
  25. 2023Evolutionary Reinforcement Learning: A Surveycited by 29
  26. 2021Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Surveycited by 28
  27. 2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 28
  28. 2021Generating and Blending Game Levels via Quality-Diversity in the Latent Space of a Variational Autoencodercited by 27
  29. 2024Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithmscited by 26
  30. 2023Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shapingcited by 25
  31. 2022Towards Deployment-Efficient Reinforcement Learning: Lower Bound and Optimalitycited by 24
  32. 2021Efficient Local Planning with Linear Function Approximationcited by 23
  33. 2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
  34. 2021When should agents explore?cited by 20
  35. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  36. 2023Generating Behaviorally Diverse Policies with Latent Diffusion Modelscited by 19
  37. 2026Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimizationcited by 19
  38. 2026The Red Queen G"odel Machine: Co-Evolving Agents and Their Evaluatorscited by 19

50 most cited of 212 All 212 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Exploration
  2. 2025DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning
  3. 2025Exploration with Foundation Models: Capabilities, Limitations, and Hybrid Approaches
  4. 2022Towards Improving Exploration in Self-Imitation Learning using Intrinsic Motivation
  5. 2026Search Inspired Exploration in Reinforcement Learning
  6. 2026Regret-Guided Search Control for Efficient Learning in AlphaZero
  7. 2024Explore-Go: Leveraging Exploration for Generalisation in Deep Reinforcement Learning
  8. 2020Guided Exploration with Proximal Policy Optimization using a Single Demonstration