mlsanityData to 7 Oct 2026

The Effectiveness of World Models for Continual Reinforcement Learning

arXiv 2211.15944 ↗First posted 29 Nov 2022cs.LG cs.AIcited by 15

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

World models power some of the most efficient reinforcement learning algorithms. In this work, we showcase that they can be harnessed for continual learning - a situation when the agent faces changing environments. World models typically employ a replay buffer for training, which can be naturally extended to continual learning. We systematically study how different selective experience replay methods affect performance, forgetting, and transfer. We also provide recommendations regarding various modeling options for using world models. The best set of choices is called Continual-Dreamer, it is task-agnostic and utilizes the world model for continual exploration. Continual-Dreamer is sample efficient and outperforms state-of-the-art task-agnostic continual reinforcement learning methods on Minigrid and Minihack benchmarks.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 54 of its 70 references that are in the map. First posted on arXiv 29 Nov 2022.

Builds on

54 of 70 references.

Closest ancestors · 3

Same direction, newest first

  1. 2020Mastering Atari with Discrete World Models5 Oct
  2. 2020Planning to Explore via Self-Supervised World Models12 May
  3. 2018World Models27 Mar

Other references · 51

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2016Learning without Forgettingcited by 2,465
  6. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  7. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  8. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,428
  9. 2016Progressive Neural Networkscited by 1,332
  10. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  11. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  12. 2017Continual Learning with Deep Generative Replaycited by 1,152
38 more
  1. 2018Experience Replay for Continual Learningcited by 1,074
  2. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  3. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  4. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  5. 2018Learning Dexterous In-Hand Manipulationcited by 643
  6. 2019Three scenarios for continual learningcited by 610
  7. 2019Gradient based sample selection for online continual learningcited by 568
  8. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  9. 2019Model-Based Reinforcement Learning for Ataricited by 495
  10. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  11. 2019Online Continual Learning with Maximally Interfered Retrievalcited by 376
  12. 2017Variational Continual Learningcited by 335
  13. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  14. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  15. 2019Self-Supervised Exploration via Disagreementcited by 263
  16. 2018Selective Experience Replay for Lifelong Learningcited by 239
  17. 2018Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselinescited by 228
  18. 2020The NetHack Learning Environmentcited by 196
  19. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 182
  20. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  21. 2022Evolving Curricula with Regret-Based Environment Designcited by 166
  22. 2020A Neural Dirichlet Process Mixture Model for Task-Free Continual Learningcited by 133
  23. 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 112
  24. 2018Measuring and regularizing networks in function spacecited by 97
  25. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 96
  26. 2018Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RLcited by 80
  27. 2021The Option Keyboard: Combining Skills in Reinforcement Learningcited by 65
  28. 2020Deep Reinforcement Learning amidst Lifelong Non-Stationaritycited by 42
  29. 2021Same State, Different Task: Continual Reinforcement Learning without Interferencecited by 42
  30. 2020Continual Model-Based Reinforcement Learning with Hypernetworkscited by 41
  31. 2021CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agentscited by 36
  32. 2020Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgettingcited by 29
  33. 2022Building a Subspace of Policies for Scalable Continual Learningcited by 28
  34. 2020Deep Reinforcement Learning for Efficient Measurement of Quantum Devicescited by 15
  35. 2022Model-based Lifelong Reinforcement Learning with Bayesian Explorationcited by 12
  36. 2020The LoCA Regret: A Consistent Metric to Evaluate Model-Based Behavior in Reinforcement Learningcited by 11
  37. 2022Towards Evaluating Adaptivity of Model-Based Reinforcement Learning Methodscited by 9
  38. 2022Reactive Exploration to Cope with Non-Stationarity in Lifelong Reinforcement Learningcited by 9

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 4; 2026: 6.

15 papers.

Sort:

By area:

  1. 2024Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglementcited by 16
  2. 2025A Survey of Continual Reinforcement Learningcited by 16
  3. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  4. 2026Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learningcited by 13
  5. 2023Multi-agent Continual Coordination via Progressive Task Contextualizationcited by 10
  6. 2025Continual Reinforcement Learning by Planning with Online World Modelscited by 8
  7. 2024Statistical Guarantees for Lifelong Reinforcement Learning using PAC-Bayes Theorycited by 7
  8. 2025Knowledge Retention for Continual Model-Based Reinforcement Learningcited by 6
  9. 2024Augmenting Replay in World Models for Continual Reinforcement Learningcited by 4
  10. 2024Towards Unraveling and Improving Generalization in World Modelscited by 3
  11. 2026Self-adapting Robotic Agents through Online Continual Reinforcement Learning with World Model Feedbackcited by 2
  12. 2026ARROW: Augmented Replay for RObust World modelscited by 1
3 more
  1. 2026The World Model Remembers, the Actor Forgets: Dream Rehearsal for Continual Model-Based RLcited by 0
  2. 2026Characterizing Replay Retention Under Dynamics Shift in Model-Based Reinforcement Learningcited by 0
  3. 2026Direct Experience World-Model Optimization: Learning the World Beyond Action Imitationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2019Continual Learning Using World Models for Pseudo-Rehearsal
  2. 2023Continual Visual Reinforcement Learning with A Life-Long World Model
  3. 2023OER: Offline Experience Replay for Continual Offline Reinforcement Learning
  4. 2025A Continual Offline Reinforcement Learning Benchmark for Navigation Tasks
  5. 2023Replay Buffer with Local Forgetting for Adapting to Local Environment Changes in Deep Model-Based Reinforcement Learning
  6. 2024Learning World Models for Unconstrained Goal Navigation
  7. 2024Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replay
  8. 2025Zero-shot World Models via Search in Memory