# Intelligent Go-Explore: Standing on the Shoulders of Giant Foundation Models

arXiv: https://arxiv.org/abs/2405.15143 · HTML: https://mlsanity.com/paper/2405.15143/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 39 references that are in the map.
- First posted on arXiv 24 May 2024.

## Builds on (most cited first, 30 of 31)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Graph of Thoughts: Solving Elaborate Problems with Large Language Models](https://mlsanity.com/paper/2308.09687/) (2023)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://mlsanity.com/paper/2309.00267/) (2023)
- [TextWorld: A Learning Environment for Text-based Games](https://mlsanity.com/paper/1806.11532/) (2018)
- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)
- [Go-Explore: a New Approach for Hard-Exploration Problems](https://mlsanity.com/paper/1901.10995/) (2019)
- [The NetHack Learning Environment](https://mlsanity.com/paper/2006.13760/) (2020)
- [Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning](https://mlsanity.com/paper/2302.02662/) (2023)
- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)
- [Neural Map: Structured Memory for Deep Reinforcement Learning](https://mlsanity.com/paper/1702.08360/) (2017)
- [Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrapping](https://mlsanity.com/paper/2402.14083/) (2024)
- [OMNI: Open-endedness via Models of human Notions of Interestingness](https://mlsanity.com/paper/2306.01711/) (2023)
- [Thought Cloning: Learning to Think while Acting by Imitating Human Thinking](https://mlsanity.com/paper/2306.00323/) (2023)
- [General Intelligence Requires Rethinking Exploration](https://mlsanity.com/paper/2211.07819/) (2022)
- [Exploration Based Language Learning for Text-Based Games](https://mlsanity.com/paper/2001.08868/) (2020)
- [A Survey of Imitation Learning Methods, Environments and Metrics](https://mlsanity.com/paper/2404.19456/) (2024)
- [Breadcrumbs to the Goal: Goal-Conditioned Exploration from Human-in-the-Loop Feedback](https://mlsanity.com/paper/2307.11049/) (2023)

## Built on by (most cited first, 30 of 33)

- [Automated Design of Agentic Systems](https://mlsanity.com/paper/2408.08435/) (2024)
- [Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents](https://mlsanity.com/paper/2505.22954/) (2025)
- [BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games](https://mlsanity.com/paper/2411.13543/) (2024)
- [SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement](https://mlsanity.com/paper/2410.20285/) (2024)
- [AgentEvolver: Towards Efficient Self-Evolving Agent System](https://mlsanity.com/paper/2511.10395/) (2025)
- [MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2503.03686/) (2025)
- [First Return, Entropy-Eliciting Explore](https://mlsanity.com/paper/2507.07017/) (2025)
- [LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities](https://mlsanity.com/paper/2504.16078/) (2025)
- [Reward Is Enough: LLMs Are In-Context Reinforcement Learners](https://mlsanity.com/paper/2506.06303/) (2025)
- [MAS-ZERO: Designing Multi-Agent Systems with Zero Supervision](https://mlsanity.com/paper/2505.14996/) (2025)
- [Weak-for-Strong: Training Weak Meta-Agent to Harness Strong Executors](https://mlsanity.com/paper/2504.04785/) (2025)
- [Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization](https://mlsanity.com/paper/2602.23008/) (2026)
- [Agent Primitives: Reusable Latent Building Blocks for Multi-Agent Systems](https://mlsanity.com/paper/2602.03695/) (2026)
- [TALES: Text Adventure Learning Environment Suite](https://mlsanity.com/paper/2504.14128/) (2025)
- [Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedback](https://mlsanity.com/paper/2410.23022/) (2024)
- [Automated Capability Discovery via Foundation Model Self-Exploration](https://mlsanity.com/paper/2502.07577/) (2025)
- [Build Your Personalized Research Group: A Multiagent Framework for Continual and Interactive Science Automation](https://mlsanity.com/paper/2510.15624/) (2025)
- [Can foundation models actively gather information in interactive environments to test hypotheses?](https://mlsanity.com/paper/2412.06438/) (2024)
- [VideoAgent: All-in-One Framework for Video Understanding and Editing](https://mlsanity.com/paper/2606.23327/) (2026)
- [Inefficiencies of Meta Agents for Agent Design](https://mlsanity.com/paper/2510.06711/) (2025)
- [AgenticRed: Evolving Agentic Systems for Red-Teaming](https://mlsanity.com/paper/2601.13518/) (2026)
- [Dual-Scale World Models for LLM Agents Towards Hard-Exploration Problems](https://mlsanity.com/paper/2509.24116/) (2025)
- [Disentangling Exploration of Large Language Models by Optimal Exploitation](https://mlsanity.com/paper/2501.08925/) (2025)
- [AutoContext: Instance-Level Context Learning for LLM Agents](https://mlsanity.com/paper/2510.02369/) (2025)
- [Toward Autonomous UI Exploration: The UIExplorer Benchmark](https://mlsanity.com/paper/2506.17779/) (2025)
- [Exploration with Foundation Models: Capabilities, Limitations, and Hybrid Approaches](https://mlsanity.com/paper/2509.19924/) (2025)
- [FOSSIL: Harnessing Feedback on Suboptimal Samples for Data-Efficient Generalisation with Imitation Learning for Embodied Vision-and-Language Tasks](https://mlsanity.com/paper/2510.11307/) (2025)
- [Cultural Alien Sampler: Open-ended art generation balancing originality and coherence](https://mlsanity.com/paper/2510.20849/) (2025)
- [Towards Scalable Oversight via Partitioned Human Supervision](https://mlsanity.com/paper/2510.22500/) (2025)
- [Joint Agent Memory and Exploration Learning via Novelty Signals](https://mlsanity.com/paper/2606.01528/) (2026)

## Abstract (arXiv)

Go-Explore is a powerful family of algorithms designed to solve hard-exploration problems built on the principle of archiving discovered states, and iteratively returning to and exploring from the most promising states. This approach has led to superhuman performance across a wide variety of challenging problems including Atari games and robotic control, but requires manually designing heuristics to guide exploration (i.e., determine which states to save and explore from, and what actions to consider next), which is time-consuming and infeasible in general. To resolve this, we propose Intelligent Go-Explore (IGE) which greatly extends the scope of the original Go-Explore by replacing these handcrafted heuristics with the intelligence and internalized human notions of interestingness captured by giant pretrained foundation models (FMs). This provides IGE with a human-like ability to instinctively identify how interesting or promising any new state is (e.g., discovering new objects, locations, or behaviors), even in complex environments where heuristics are hard to define. Moreover, IGE offers the exciting opportunity to recognize and capitalize on serendipitous discoveries -- states encountered during exploration that are valuable in terms of exploration, yet where what makes them interesting was not anticipated by the human user. We evaluate our algorithm on a diverse range of language and vision-based tasks that require search and exploration. Across these tasks, IGE strongly exceeds classic reinforcement learning and graph search baselines, and also succeeds where prior state-of-the-art FM agents like Reflexion completely fail. Overall, Intelligent Go-Explore combines the tremendous strengths of FMs and the powerful Go-Explore algorithm, opening up a new frontier of research into creating more generally capable agents with impressive exploration capabilities.