mlsanityData to 7 Oct 2026

Dreaming of Many Worlds: Learning Contextual World Models Aids Zero-Shot Generalization

arXiv 2403.10967 ↗First posted 16 Mar 2024cs.LG cs.AIcited by 16

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Zero-shot generalization (ZSG) to unseen dynamics is a major challenge for creating generally capable embodied agents. To address the broader challenge, we start with the simpler setting of contextual reinforcement learning (cRL), assuming observability of the context values that parameterize the variation in the system's dynamics, such as the mass or dimensions of a robot, without making further simplifying assumptions about the observability of the Markovian state. Toward the goal of ZSG to unseen variation in context, we propose the contextual recurrent state-space model (cRSSM), which introduces changes to the world model of Dreamer (v3) (Hafner et al., 2023). This allows the world model to incorporate context for inferring latent Markovian states from the observations and modeling the latent dynamics. Our approach is evaluated on two tasks from the CARL benchmark suite, which is tailored to study contextual RL. Our experiments show that such systematic incorporation of the context improves the ZSG of the policies trained on the "dreams" of the world model. We further find qualitatively that our approach allows Dreamer to disentangle the latent state from context, allowing it to extrapolate its dreams to the many worlds of unseen contexts. The code for all our experiments is available at https://github.com/sai-prasanna/dreaming_of_many_worlds.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 37 of its 43 references that are in the map. First posted on arXiv 16 Mar 2024.

Builds on

37 of 43 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023TD-MPC2: Scalable, Robust World Models for Continuous Control25 Oct
  2. 2023Mastering Diverse Domains through World Models10 Jan
  3. 2022DayDreamer: World Models for Physical Robot Learning28 Jun
  4. 2021Procedural Generalization by Planning with Self-Supervised World Models2 Nov
  5. 2021Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment12 Apr
  6. 2020Mastering Atari with Discrete World Models5 Oct
  7. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
  8. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models30 May

Other references · 29

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  3. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  4. 2018DeepMind Control Suitecited by 935
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  6. 2018Learning Dexterous In-Hand Manipulationcited by 643
  7. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  8. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  9. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  10. 2016Learning to reinforcement learncited by 369
  11. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
  12. 2015Contextual Markov Decision Processescited by 213
17 more
  1. 2021Towards Causal Representation Learningcited by 207
  2. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  3. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  4. 2020Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learningcited by 95
  5. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  6. 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
  7. 2017Markov Decision Processes with Continuous Side Informationcited by 54
  8. 2022Contextualize Me -- The Case for Context in Reinforcement Learningcited by 41
  9. 2021The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priorscited by 40
  10. 2022Transformers are Meta-Reinforcement Learnerscited by 39
  11. 2021Block Contextual MDPs for Continual Learningcited by 25
  12. 2020Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trialscited by 20
  13. 2023Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policiescited by 18
  14. 2022The Benefits of Model-Based Generalization in Reinforcement Learningcited by 17
  15. 2022A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learningcited by 15
  16. 2020Zero-Shot Terrain Generalization for Visual Locomotion Policiescited by 13
  17. 2023Dream to Adapt: Meta Reinforcement Learning by Latent Context Imagination and MDP Imaginationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 5.

16 papers.

Sort:

By area:

  1. 2025Is Single-View Mesh Reconstruction Ready for Robotics?cited by 6
  2. 2026A Mechanistic View on Video Generation as World Models: State and Dynamicscited by 6
  3. 2025Dynamics-Aligned Latent Imagination in Contextual World Models for Zero-Shot Generalizationcited by 5
  4. 2024Craftium: Bridging Flexibility and Efficiency for Rich 3D Single- and Multi-Agent Environmentscited by 4
  5. 2025Out-of-Distribution Generalization with a SPARC: Racing 100 Unseen Vehicles with a Single Policycited by 4
  6. 2026MetaMind: General and Cognitive World Models in Multi-Agent Systems by Meta-Theory of Mindcited by 3
  7. 2024GRAM: Generalization in Deep RL with a Robust Adaptation Modulecited by 2
  8. 2025Adaptive Shielding for Safe Reinforcement Learning under Hidden-Parameter Dynamics Shiftscited by 2
  9. 2025Efficient Generation of Diverse Cooperative Agents with World Modelscited by 1
  10. 2026Contextual Intelligence The Next Leap for Reinforcement Learningcited by 1
  11. 2024Zero-shot Generalization in Inventory Management: Train, then Estimate and Decidecited by 0
  12. 2025Zero-Shot Context Generalization in Reinforcement Learning from Few Training Contextscited by 0
4 more
  1. 2025Context-Aware Model-Based Reinforcement Learning for Autonomous Racingcited by 0
  2. 2025Vehicle Dynamics Embedded World Models for Autonomous Drivingcited by 0
  3. 2026Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learningcited by 0
  4. 2026Dynamic Context Scheduling: Learning Beyond the Static Universecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Zero-shot World Models via Search in Memory
  2. 2026Towards Zero-Shot Task Transfer with Neurosymbolic World Models
  3. 2026Reinforcing VLAs in Task-Agnostic World Models
  4. 2025Dream to Generalize: Zero-Shot Model-Based Reinforcement Learning for Unseen Visual Distractions
  5. 2025Multimodal Dreaming: A Global Workspace Approach to World Model-Based Reinforcement Learning
  6. 2021Latent Imagination Facilitates Zero-Shot Transfer in Autonomous Racing
  7. 2021DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representations
  8. 2018World Models