mlsanityData to 14 Oct 2026

Multitask Adaptation by Retrospective Exploration with Learned World Models

arXiv 2110.13241 ↗First posted 25 Oct 2021cs.LGcited by 0

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Model-based reinforcement learning (MBRL) allows solving complex tasks in a sample-efficient manner. However, no information is reused between the tasks. In this work, we propose a meta-learned addressing model called RAMa that provides training samples for the MBRL agent taken from continuously growing task-agnostic storage. The model is trained to maximize the expected agent's performance by selecting promising trajectories solving prior tasks from the storage. We show that such retrospective exploration can accelerate the learning process of the MBRL agent by better informing learned dynamics and prompting agent with exploratory trajectories. We test the performance of our approach on several domains from the DeepMind control suite, from Metaworld multitask benchmark, and from our bespoke environment implemented with a robotic NVIDIA Isaac simulator to test the ability of the model to act in a photorealistic, ray-traced environment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 28 of its 38 references that are in the map. First posted on arXiv 25 Oct 2021.

Builds on

28 of 38 references.

Closest ancestors · 4

Same direction, newest first

  1. 2020Planning to Explore via Self-Supervised World Models12 May
  2. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
  3. 2019When to Trust Your Model: Model-Based Policy Optimization19 Jun
  4. 2018Recurrent World Models Facilitate Policy Evolution4 Sep

Other references · 24

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  5. 2013Auto-Encoding Variational Bayescited by 4,053
  6. 2015Trust Region Policy Optimizationcited by 2,580
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  8. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  9. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  10. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  11. 2017Hindsight Experience Replaycited by 1,136
  12. 2018Learning to Reweight Examples for Robust Deep Learningcited by 675
12 more
  1. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  2. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 429
  3. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  4. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  5. 2017Neural Episodic Controlcited by 151
  6. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 117
  7. 2019Data Valuation using Reinforcement Learningcited by 116
  8. 2016Model-Free Episodic Controlcited by 102
  9. 2017Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learningcited by 98
  10. 2018Unsupervised Predictive Memory in a Goal-Directed Agentcited by 55
  11. 2019Sub-policy Adaptation for Hierarchical Reinforcement Learningcited by 47
  12. 2017Variational Memory Addressing in Generative Modelscited by 17

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023HarmonyDream: Task Harmonization Inside World Models
  2. 2024Hierarchical Meta-Reinforcement Learning via Automated Macro-Action Discovery
  3. 2024Boosting Hierarchical Reinforcement Learning with Meta-Learning for Complex Task Adaptation
  4. 2024Sample Efficient Myopic Exploration Through Multitask Reinforcement Learning with Diverse Tasks
  5. 2025Knowledge Transfer in Model-Based Reinforcement Learning Agents for Efficient Multi-Task Learning
  6. 2026Meta-Learning and Meta-Reinforcement Learning -- Tracing the Path towards DeepMind's Adaptive Agent
  7. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers
  8. 2026Knowledge Reutilization in Meta-Reinforcement Learning