# What Should World Models Forget? Stratified Retention for Continual Adaptation

arXiv: https://arxiv.org/abs/2610.03713 · HTML: https://mlsanity.com/paper/2610.03713/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 18 of its 24 references that are in the map.
- First posted on arXiv 2 Oct 2026.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)

## Builds on (most cited first, 18 of 18)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Titans: Learning to Memorize at Test Time](https://mlsanity.com/paper/2501.00663/) (2024)
- [LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics](https://mlsanity.com/paper/2511.08544/) (2025)
- [Training Agents Inside of Scalable World Models](https://mlsanity.com/paper/2509.24527/) (2025)
- [Don't forget, there is more than forgetting: new metrics for Continual Learning](https://mlsanity.com/paper/1810.13166/) (2018)
- [Nested Learning: The Illusion of Deep Learning Architectures](https://mlsanity.com/paper/2512.24695/) (2025)
- [stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluation](https://mlsanity.com/paper/2605.21800/) (2026)
- [The Effectiveness of World Models for Continual Reinforcement Learning](https://mlsanity.com/paper/2211.15944/) (2022)
- [Towards Evaluating Adaptivity of Model-Based Reinforcement Learning Methods](https://mlsanity.com/paper/2204.11464/) (2022)
- [Knowledge Retention for Continual Model-Based Reinforcement Learning](https://mlsanity.com/paper/2503.04256/) (2025)
- [When Benchmarks Age: Temporal Misalignment through Large Language Model Factuality Evaluation](https://mlsanity.com/paper/2510.07238/) (2025)
- [Augmenting Replay in World Models for Continual Reinforcement Learning](https://mlsanity.com/paper/2401.16650/) (2024)
- [LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics](https://mlsanity.com/paper/2608.27395/) (2026)
- [Holistic Continual Learning under Concept Drift with Adaptive Memory Realignment](https://mlsanity.com/paper/2507.02310/) (2025)
- [The World Model Remembers, the Actor Forgets: Dream Rehearsal for Continual Model-Based RL](https://mlsanity.com/paper/2607.19749/) (2026)
- [Evolving Machine Learning in Non-Stationary Environments: A Unified Survey of Drift, Forgetting, and Adaptation](https://mlsanity.com/paper/2505.17902/) (2025)

## Abstract (arXiv)

Continual learning treats degradation on previously seen data as evidence of failure, a convention inherited from settings with a stationary prediction target, where a correct label remains correct indefinitely. World models do not satisfy this condition. Their prediction target is the environment, which changes, so knowledge that was accurate when acquired may later become false, and discarding it is required behavior rather than a defect. Non-stationary ground truth is well studied in the concept drift literature and in the temporal factuality of language models, but has not been formulated for world models, which are distinctive in that they also encode knowledge that must never be revised. We argue that continual world models require retention stratified by invariance timescale, separating invariants such as physics and object permanence, which must never be revised, from instance-level facts that should be revised as soon as the environment changes. Standard forgetting metrics cannot distinguish a world model that has correctly revised outdated knowledge from one that has suffered catastrophic forgetting, and consequently rank a frozen model highest, while existing physical-reasoning benchmarks evaluate only frozen checkpoints. We propose differential retention, which reports invariant regression testing across the adaptation stream jointly with revision latency, without aggregation.