mlsanityData to 14 Oct 2026

Continual Reinforcement Learning by Planning with Online World Models

arXiv 2507.09177 ↗First posted 12 Jul 2025cs.LG cs.AI stat.MLcited by 8

On the map: World models → Latent dynamics models and model-based RL

also in 1 more
Abstract (arXiv)

Continual reinforcement learning (CRL) refers to a naturalistic setting where an agent needs to endlessly evolve, by trial and error, to solve multiple tasks that are presented sequentially. One of the largest obstacles to CRL is that the agent may forget how to solve previous tasks when learning a new task, known as catastrophic forgetting. In this paper, we propose to address this challenge by planning with online world models. Specifically, we learn a Follow-The-Leader shallow model online to capture the world dynamics, in which we plan using model predictive control to solve a set of tasks specified by any reward functions. The online world model is immune to forgetting by construction with a proven regret bound of O(√K2Dlog(T)) under mild assumptions. The planner searches actions solely based on the latest online model, thus forming a FTL Online Agent (OA) that updates incrementally. To assess OA, we further design Continual Bench, a dedicated environment for CRL, and compare with several strong baselines under the same model-planning algorithmic framework. The empirical results show that OA learns continuously to solve new tasks while not forgetting old skills, outperforming agents built on deep world models with various continual learning techniques.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 38 of its 56 references that are in the map. First posted on arXiv 12 Jul 2025.

Builds on

38 of 56 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Locality Sensitive Sparse Encoding for Learning World Models Online23 Jan
  2. 2022The Effectiveness of World Models for Continual Reinforcement Learning29 Nov
  3. 2022Curious Exploration via Structured World Models Yields Zero-Shot Object Manipulation22 Jun
  4. 2019Exploring Model-based Planning with Policy Networks20 Jun
  5. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models30 May

Other references · 33

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  4. 2018Experience Replay for Continual Learningcited by 1,095
  5. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  6. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  7. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  8. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  9. 2019Gradient based sample selection for online continual learningcited by 571
  10. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 521
  11. 2018Progress & Compress: A scalable framework for continual learningcited by 520
  12. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
21 more
  1. 2019Orthogonal Gradient Descent for Continual Learningcited by 369
  2. 2019On Tiny Episodic Memories in Continual Learningcited by 307
  3. 2018Selective Experience Replay for Lifelong Learningcited by 241
  4. 2020The NetHack Learning Environmentcited by 198
  5. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 124
  6. 2022Class-Incremental Continual Learning into the eXtended DER-versecited by 124
  7. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 121
  8. 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 116
  9. 2020Sample-efficient Cross-Entropy Method for Real-time Planningcited by 95
  10. 2018Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RLcited by 81
  11. 2018Selfless Sequential Learningcited by 44
  12. 2020Continual Model-Based Reinforcement Learning with Hypernetworkscited by 44
  13. 2021Same State, Different Task: Continual Reinforcement Learning without Interferencecited by 43
  14. 2021MBRL-Lib: A Modular Library for Model-based Reinforcement Learningcited by 42
  15. 2023The Ideal Continual Learner: An Agent That Never Forgetscited by 35
  16. 2021Autonomous Reinforcement Learning: Formalism and Benchmarkingcited by 33
  17. 2022Building a Subspace of Policies for Scalable Continual Learningcited by 28
  18. 2019A Meta-MDP Approach to Exploration for Lifelong Reinforcement Learningcited by 18
  19. 2023Block Coordinate Descent on Smooth Manifolds: Convergence Theory and Twenty-One Examplescited by 16
  20. 2023Continual Task Allocation in Meta-Policy Network via Sparse Promptingcited by 11
  21. 2023Forget-free Continual Learning with Soft-Winning SubNetworkscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  2. 2026A Mechanistic View on Video Generation as World Models: State and Dynamicscited by 6
  3. 2026Self-evolving Embodied AIcited by 2
  4. 2026Learning Task-Sufficient World Models by Synergizing Agentic Exploration and Structured Modelingcited by 2
  5. 2026Measuring the Value of World-Model Updates: A Counterfactual Utility Protocol for Continual Adaptationcited by 1
  6. 2026Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learningcited by 0
  7. 2026ProgAgent:A Continual RL Agent with Progress-Aware Rewardscited by 0
  8. 2026CE4L: Continual Ego, Exo, and Ego-Exo Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking the Foundations for Continual Reinforcement Learning
  2. 2025A Survey of Continual Reinforcement Learning
  3. 2026Position: Deployed Reinforcement Learning should be Continual
  4. 2024Continual Task Learning through Adaptive Policy Self-Composition
  5. 2025The Cell Must Go On: Agar.io for Continual Reinforcement Learning
  6. 2025Advancements and Challenges in Continual Reinforcement Learning: A Comprehensive Review
  7. 2024Augmenting Replay in World Models for Continual Reinforcement Learning
  8. 2024Reset & Distill: A Recipe for Overcoming Negative Transfer in Continual Reinforcement Learning