Data to 14 Oct 2026

Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment

arXiv 2104.05632 ↗First posted 12 Apr 2021cs.LG cs.AIcited by 42

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Reinforcement learning from large-scale offline datasets provides us with the ability to learn policies without potentially unsafe or impractical exploration. Significant progress has been made in the past few years in dealing with the challenge of correcting for differing behavior between the data collection and learned policies. However, little attention has been paid to potentially changing dynamics when transferring a policy to the online setting, where performance can be up to 90% reduced for existing methods. In this paper we address this problem with Augmented World Models (AugWM). We augment a learned dynamics model with simple transformations that seek to capture potential changes in physical properties of the robot, leading to more robust policies. We not only train our policy in this new setting, but also provide it with the sampled augmentation as a context, allowing it to adapt to changes in the environment. At test time we learn the context in a self-supervised fashion by approximating the augmentation which corresponds to the new environment. We rigorously evaluate our approach on over 100 different changed dynamics settings, and show that this simple approach can significantly improve the zero-shot generalization of a recent state-of-the-art baseline, often achieving successful policies where the baseline fails.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 53 of its 59 references that are in the map. First posted on arXiv 12 Apr 2021.

Builds on

53 of 59 references.

Closest ancestors · 5

Same direction, newest first

Other references · 48

2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
2016OpenAI Gymcited by 1,736
2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
2019Solving Rubik's Cube with a Robot Handcited by 734

36 more

2018Learning Dexterous In-Hand Manipulationcited by 687
2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
2019Behavior Regularized Offline Reinforcement Learningcited by 662
2020MOPO: Model-based Offline Policy Optimizationcited by 636
2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
2019Model-Based Reinforcement Learning for Ataricited by 502
2018Generalizing to Unseen Domains via Adversarial Data Augmentationcited by 451
2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
2019Challenges of Real-World Reinforcement Learningcited by 291
2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 276
2015Contextual Markov Decision Processescited by 220
2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 217
2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
2016Continuous Deep Q-Learning with Model-based Accelerationcited by 157
2017Preparing for the Unknown: Learning a Universal Policy with Online System Identificationcited by 141
2020Model-Based Offline Planningcited by 129
2020Self-Supervised Policy Adaptation during Deploymentcited by 123
2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 101
2020Counterfactual Data Augmentation using Locally Factored Dynamicscited by 93
2019Observational Overfitting in Reinforcement Learningcited by 91
2018Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RLcited by 81
2020Transient Non-Stationarity and Generalisation in Deep Reinforcement Learningcited by 75
2020Automatic Data Augmentation for Generalization in Deep Reinforcement Learningcited by 71
2013Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizationscited by 70
2017Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Taskcited by 66
2020Meta-Learning Requires Meta-Augmentationcited by 57
2017Markov Decision Processes with Continuous Side Informationcited by 56
2019Environment Probing Interaction Policiescited by 39
2020Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learningcited by 35
2020Ready Policy One: World Building Through Active Learningcited by 33
2017Robust and Efficient Transfer Learning with Hidden-Parameter Markov Decision Processescited by 29
2020Exploration in Approximate Hyper-State Space for Meta Reinforcement Learningcited by 28
2017Reinforcement Learning for Pivoting Taskcited by 17
2017Benchmark Environments for Multitask Learning in Continuous Domainscited by 14
2021Linear Representation Meta-Reinforcement Learning for Instant Adaptationcited by 9

Built on by

Cited by year

Citing papers per year: 2021: 5; 2022: 14; 2023: 2; 2024: 9; 2025: 12.

42 papers.

Sort:

By area:

2022Evolving Curricula with Regret-Based Environment Designcited by 173
2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
2023Synthetic Experience Replaycited by 101
2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observationscited by 63
2021Revisiting Design Choices in Offline Model-Based Reinforcement Learningcited by 50
2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 41
2024Policy-Guided Diffusioncited by 41
2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learningcited by 23
2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Beliefcited by 22
2024GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learningcited by 22
2023Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policiescited by 20

30 more

2022On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrationscited by 19
2021Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functionscited by 17
2024Dreaming of Many Worlds: Learning Contextual World Models Aids Zero-Shot Generalizationcited by 16
2021Generalization of Reinforcement Learning with Policy-Aware Adversarial Data Augmentationcited by 15
2022A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learningcited by 15
2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
2022Representation Learning in Deep RL via Discrete Information Bottleneckcited by 13
2025State-Covering Trajectory Stitching for Diffusion Plannerscited by 13
2022Learning General World Models in a Handful of Reward-Free Deploymentscited by 12
2024Hindsight PRIORs for Reward Learning from Human Preferencescited by 12
2022Learn the Time to Learn: Replay Scheduling in Continual Learningcited by 11
2025Adapting World Models with Latent-State Dynamics Residualscited by 9
2024The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learningcited by 8
2024Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learningcited by 7
2025Is Single-View Mesh Reconstruction Ready for Robotics?cited by 6
2025Zero-Shot Reinforcement Learning Under Partial Observabilitycited by 6
2025Dynamics-Aligned Latent Imagination in Contextual World Models for Zero-Shot Generalizationcited by 5
2025Imagined Autocurriculacited by 4
2022Prototypical context-aware dynamics generalization for high-dimensional model-based reinforcement learningcited by 3
2024Skills Regularized Task Decomposition for Multi-task Offline Reinforcement Learningcited by 3
2025PrivilegedDreamer: Explicit Imagination of Privileged Information for Rapid Adaptation of Learned Policiescited by 3
2022Open-Ended Diverse Solution Discovery with Regulated Behavior Patterns for Cross-Domain Adaptationcited by 1
2024Ensemble Successor Representations for Task Generalization in Offline-to-Online Reinforcement Learningcited by 1
2025On the Fly Adaptation of Behavior Tree-Based Policies through Reinforcement Learningcited by 1
2021Dream to Explore: Adaptive Simulations for Autonomous Systemscited by 0
2022Model Generation with Provable Coverability for Offline Reinforcement Learningcited by 0
2024One-shot World Models Using a Transformer Trained on a Synthetic Priorcited by 0
2025Zero-Shot Context Generalization in Reinforcement Learning from Few Training Contextscited by 0
2025Pruning Cannot Hurt Robustness: Certified Trade-offs in Reinforcement Learningcited by 0
2025Vehicle Dynamics Embedded World Models for Autonomous Drivingcited by 0

Similar, not cited

Close in wording, no citation either way