mlsanityData to 14 Oct 2026

Characterizing Replay Retention Under Dynamics Shift in Model-Based Reinforcement Learning

arXiv 2609.18167 ↗First posted 16 Sep 2026cs.RO cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Adapting to changes in robot dynamics requires learning from new data without discarding experience that may still be useful. In continual model-based reinforcement learning (RL), replay collected before a dynamics change can slow adaptation, while removing it unnecessarily reduces available training data and can be especially costly if earlier dynamics return. We study when recent transitions are preferable to the full replay history. Two quantities characterize this trade-off: change magnitude and age-staleness area under the curve (AUC), measuring how well transition age separates stale from fresh data. Forgetting stale data helps after large permanent shifts but hurts when dynamics recur and older data becomes useful again. Choosing a replay strategy therefore depends on predicting when older data will help or hurt. We test these effects across two locomotion morphologies, two model-based RL algorithms, and Real-World RL benchmark perturbations. Because ground-truth staleness labels are unavailable on deployed robots, we evaluate whether an estimator built from interaction data can still provide the quantities needed to choose a replay strategy after permanent changes. Our results show that replay retention depends on change magnitude and on how the dynamics evolve.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 24 of its 35 references that are in the map. First posted on arXiv 16 Sep 2026.

Builds on

24 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Augmenting Replay in World Models for Continual Reinforcement Learning30 Jan

Other references · 23

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  2. 2023Mastering Diverse Domains through World Modelscited by 1,253
  3. 2015Prioritized Experience Replaycited by 1,099
  4. 2018DeepMind Control Suitecited by 956
  5. 2017Deep Reinforcement Learning that Matterscited by 754
  6. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  7. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  8. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
  9. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  10. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 276
  11. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  12. 2018Selective Experience Replay for Lifelong Learningcited by 241
11 more
  1. 2014Robots that can adapt like animalscited by 204
  2. 2020Revisiting Fundamentals of Experience Replaycited by 175
  3. 2017A Deeper Look at Experience Replaycited by 105
  4. 2020Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learningcited by 98
  5. 2020An empirical investigation of the challenges of real-world reinforcement learningcited by 77
  6. 2020Experience Replay with Likelihood-free Importance Weightscited by 42
  7. 2021Minimum-Delay Adaptation in Non-Stationary Reinforcement Learning via Online High-Confidence Change-Point Detectioncited by 18
  8. 2022The Effectiveness of World Models for Continual Reinforcement Learningcited by 17
  9. 2023Curious Replay for Model-based Adaptationcited by 16
  10. 2023Novelty Detection in Reinforcement Learning with World Modelscited by 8
  11. 2023Replay Buffer with Local Forgetting for Adapting to Local Environment Changes in Deep Model-Based Reinforcement Learningcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning
  2. 2026Data-Free Continual Learning for World Action Model with Rectified Generative Replay
  3. 2026Changepoint-Aware World Models: Detecting Dynamics Shifts and Recovering by Forgetting Stale Replay in Model-Based RL
  4. 2026RIFAR: Reliability and Forgetting-Aware Replay for Continual Robot Learning
  5. 2026Memory Retention Is Not Enough to Master Memory Tasks in Reinforcement Learning
  6. 2026World Action Models Enable Continual Imitation Learning with Recurrent Generative Replays
  7. 2026Dynamics Are Learned, Not Told: Semi-Supervised Discovery of Latent Dynamics Geometries For Zero-Shot Policy Adaptation
  8. 2024Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replay