mlsanityData to 7 Oct 2026

The World Model Remembers, the Actor Forgets: Dream Rehearsal for Continual Model-Based RL

arXiv 2607.19749 ↗First posted 22 Jul 2026cs.LG cs.AIcited by 0

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Model-based reinforcement-learning agents of the DreamerV3 family forget catastrophically when trained on task sequences, even when an unbounded replay buffer preserves every earlier experience. We ask a question the continual-RL literature has assumed an answer to but never measured: which component forgets? Under never-clear replay, pre-registered component-level probes (n=3 seeds throughout) show that the world model retains essentially everything measurable about old tasks -- reward discrimination (retention ratio ~1.0), value estimates, and termination structure -- while the actor's behavior collapses. Forgetting in this regime is a channel problem, not a memory problem. We demonstrate this by intervention: with the world model frozen and identical imagined rollouts, reinforcement learning in imagination fails to recover a lost skill (0/3 seeds), while supervised self-imitation on the world model's own graded dreams recovers it on 3/3 seeds with zero environment interaction. Interleaved during training, this graded dream rehearsal yields a task-label-free, parameter-constant continual learner: 3/3 four-task chains retained where plain replay passes 0/3, 3/3 eight-task chains, and consistent gains over matched real-episode cloning (paired difference +0.13, bootstrap 95% CI [0.07, 0.24], complete seed separation). The dream-grading step is load-bearing: we characterize two scoring failure modes, provide an offline selection gauge that caught both before they contaminated results, and give a realized-first grading rule that closes them. All experiments were pre-registered with committed protocols; every refuted hypothesis is reported.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 13 of its 13 references that are in the map. First posted on arXiv 22 Jul 2026.

Builds on

13 of 13 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026ARROW: Augmented Replay for RObust World models12 Mar
  2. 2024Augmenting Replay in World Models for Continual Reinforcement Learning30 Jan
  3. 2023Mastering Diverse Domains through World Models10 Jan
  4. 2022The Effectiveness of World Models for Continual Reinforcement Learning29 Nov

Other references · 9

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2016Progressive Neural Networkscited by 1,332
  3. 2017Continual Learning with Deep Generative Replaycited by 1,152
  4. 2018Experience Replay for Continual Learningcited by 1,074
  5. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  6. 2018Self-Imitation Learningcited by 135
  7. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  8. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  9. 2026World Action Models Enable Continual Imitation Learning with Recurrent Generative Replayscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learning
  2. 2025Knowledge Retention for Continual Model-Based Reinforcement Learning
  3. 2025Memento 2: Learning by Stateful Reflective Memory
  4. 2026COUNTERMEM: World-Model Verified Counter-Factual Memory for Language Agents
  5. 2025Continual Reinforcement Learning by Planning with Online World Models
  6. 2026Benchmarking World Models for Continual Learning on Compositional Tasks
  7. 2026Can Vision-Language-Action Models Learn from Real-World Data Continually without Forgetting?
  8. 2024Mastering Memory Tasks with World Models