Preemptive Solving of Future Problems: Multitask Preplay in Humans and Machines
Not on the map. Why?
Abstract (arXiv)
Humans can pursue a near-infinite variety of tasks, but typically can only pursue a small number at the same time. We hypothesize that humans leverage experience on one task to preemptively learn solutions to other tasks that were accessible but not pursued. We formalize this idea as Multitask Preplay, a novel algorithm that replays experience on one task as the starting point for "preplay" -- counterfactual simulation of an accessible but unpursued task. Preplay is used to learn a predictive representation that can support fast, adaptive task performance later on. We first show that, compared to traditional planning and predictive representation methods, multitask preplay better predicts how humans generalize to tasks that were accessible but not pursued in a small grid-world, even when people didn't know they would need to generalize to these tasks. We then show these predictions generalize to Craftax, a partially observable 2D Minecraft environment. Finally, we show that Multitask Preplay enables artificial agents to learn behaviors that transfer to novel Craftax worlds sharing task co-occurrence structure. These findings demonstrate that Multitask Preplay is a scalable theory of how humans counterfactually learn and generalize across multiple tasks; endowing artificial agents with the same capacity can significantly improve their performance in challenging multitask environments.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 28 of its 50 references that are in the map. First posted on arXiv 8 Jul 2025.
Builds on
28 of 50 references.
References · 28
- 2017Attention Is All You Needcited by 60,361
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
- 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
- 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
- 2017Hindsight Experience Replaycited by 1,136
- 2015Prioritized Experience Replaycited by 1,099
- 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
- 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
- 2016Successor Features for Transfer in Reinforcement Learningcited by 340
- 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
16 more
- 2020The NetHack Learning Environmentcited by 198
- 2021Open-Ended Learning Leads to Generally Capable Agentscited by 182
- 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
- 2021Discovering and Achieving Goals via World Modelscited by 131
- 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
- 2022Discovered Policy Optimisationcited by 104
- 2018Universal Successor Features Approximatorscited by 103
- 2019Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvementcited by 88
- 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 88
- 2019Exploration via Hindsight Goal Generationcited by 54
- 2018Many-Goals Reinforcement Learningcited by 22
- 2021MHER: Model-based Hindsight Experience Replaycited by 15
- 2025Improving Transformer World Models for Data-Efficient RLcited by 13
- 2019Directed Exploration for Reinforcement Learningcited by 9
- 2020Reinforcement Learning for Sparse-Reward Object-Interaction Tasks in a First-person Simulated 3D Environmentcited by 8
- 2024Predictive representations: building blocks of intelligencecited by 4
Built on by
Citing papers per year: 2025: 2.
2 papers.