# Rapid Task-Solving in Novel Environments

arXiv: https://arxiv.org/abs/2006.03662 · HTML: https://mlsanity.com/paper/2006.03662/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jun 2020.

## Built on by (most cited first, 23 of 23)

- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [TransDreamer: Reinforcement Learning with Transformer World Models](https://mlsanity.com/paper/2202.09481/) (2022)
- [RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation](https://mlsanity.com/paper/2306.11706/) (2023)
- [Transformers are Meta-Reinforcement Learners](https://mlsanity.com/paper/2206.06614/) (2022)
- [Towards mental time travel: a hierarchical memory for reinforcement learning agents](https://mlsanity.com/paper/2105.14039/) (2021)
- [Large-Scale Retrieval for Reinforcement Learning](https://mlsanity.com/paper/2206.05314/) (2022)
- [Teaching Agents how to Map: Spatial Reasoning for Multi-Object Navigation](https://mlsanity.com/paper/2107.06011/) (2021)
- [Recurrent Hypernetworks are Surprisingly Strong in Meta-RL](https://mlsanity.com/paper/2309.14970/) (2023)
- [MemoNav: Working Memory Model for Visual Navigation](https://mlsanity.com/paper/2402.19161/) (2024)
- [Diffusion Models as Optimizers for Efficient Planning in Offline RL](https://mlsanity.com/paper/2407.16142/) (2024)
- [Graph schemas as abstractions for transfer learning, inference, and planning](https://mlsanity.com/paper/2302.07350/) (2023)
- [Disentangling Abstraction from Statistical Pattern Matching in Human and Machine Learning](https://mlsanity.com/paper/2204.01437/) (2022)
- [Generalization Properties of Retrieval-based Models](https://mlsanity.com/paper/2210.02617/) (2022)
- [On learning history based policies for controlling Markov decision processes](https://mlsanity.com/paper/2211.03011/) (2022)
- [Learning Dynamic Cognitive Map with Autonomous Navigation](https://mlsanity.com/paper/2411.08447/) (2024)
- [SplAgger: Split Aggregation for Meta-Reinforcement Learning](https://mlsanity.com/paper/2403.03020/) (2024)
- [TransDreamerV3: Implanting Transformer In DreamerV3](https://mlsanity.com/paper/2506.17103/) (2025)
- [How to Learn and Represent Abstractions: An Investigation using Symbolic Alchemy](https://mlsanity.com/paper/2112.08360/) (2021)
- [MemoNav: Selecting Informative Memories for Visual Navigation](https://mlsanity.com/paper/2208.09610/) (2022)
- [Building spatial world models from sparse transitional episodic memories](https://mlsanity.com/paper/2505.13696/) (2025)
- [From Memories to Maps: Mechanisms of In-Context Reinforcement Learning in Transformers](https://mlsanity.com/paper/2506.19686/) (2025)
- [Fast exploration and learning of latent graphs with aliased observations](https://mlsanity.com/paper/2303.07397/) (2023)
- [Caching for the Future: Scrub Jay Episodic Memory Principles for Agent Memory Systems](https://mlsanity.com/paper/2608.04746/) (2026)

## Abstract (arXiv)

We propose the challenge of rapid task-solving in novel environments (RTS), wherein an agent must solve a series of tasks as rapidly as possible in an unfamiliar environment. An effective RTS agent must balance between exploring the unfamiliar environment and solving its current task, all while building a model of the new environment over which it can plan when faced with later tasks. While modern deep RL agents exhibit some of these abilities in isolation, none are suitable for the full RTS challenge. To enable progress toward RTS, we introduce two challenge domains: (1) a minimal RTS challenge called the Memory&Planning Game and (2) One-Shot StreetLearn Navigation, which introduces scale and complexity from real-world data. We demonstrate that state-of-the-art deep RL agents fail at RTS in both domains, and that this failure is due to an inability to plan over gathered knowledge. We develop Episodic Planning Networks (EPNs) and show that deep-RL agents with EPNs excel at RTS, outperforming the nearest baseline by factors of 2-3 and learning to navigate held-out StreetLearn maps within a single episode. We show that EPNs learn to execute a value iteration-like planning algorithm and that they generalize to situations beyond their training experience. algorithm and that they generalize to situations beyond their training experience.