# Self-adapting Robotic Agents through Online Continual Reinforcement Learning with World Model Feedback

arXiv: https://arxiv.org/abs/2603.04029 · HTML: https://mlsanity.com/paper/2603.04029/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Latent dynamics models and model-based RL, one of 11 directions in World models.
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 4 Mar 2026.

## Placed in

- World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/)
- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 15 of 15)

- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [TD-MPC2: Scalable, Robust World Models for Continuous Control](https://mlsanity.com/paper/2310.16828/) (2023)
- [Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning](https://mlsanity.com/paper/1803.11347/) (2018)
- [Towards Continual Reinforcement Learning: A Review and Perspectives](https://mlsanity.com/paper/2012.13490/) (2020)
- [Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL](https://mlsanity.com/paper/1812.07671/) (2018)
- [Task-Agnostic Online Reinforcement Learning with an Infinite Mixture of Gaussian Processes](https://mlsanity.com/paper/2006.11441/) (2020)
- [The Effectiveness of World Models for Continual Reinforcement Learning](https://mlsanity.com/paper/2211.15944/) (2022)
- [Novelty Detection in Reinforcement Learning with World Models](https://mlsanity.com/paper/2310.08731/) (2023)
- [Statistical Context Detection for Deep Lifelong Reinforcement Learning](https://mlsanity.com/paper/2405.19047/) (2024)
- [World Models for Anomaly Detection during Model-Based Reinforcement Learning Inference](https://mlsanity.com/paper/2503.02552/) (2025)
- [A Safe Control Architecture Based on a Model Predictive Control Supervisor for Autonomous Driving](https://mlsanity.com/paper/2201.13298/) (2022)
- [Advancements and Challenges in Continual Reinforcement Learning: A Comprehensive Review](https://mlsanity.com/paper/2506.21899/) (2025)

## Built on by (most cited first, 2 of 2)

- [Measuring the Value of World-Model Updates: A Counterfactual Utility Protocol for Continual Adaptation](https://mlsanity.com/paper/2609.10954/) (2026)
- [Reinforcement Learning with Decomposed Subtasks](https://mlsanity.com/paper/2609.27035/) (2026)

## Abstract (arXiv)

As learning-based robotic controllers are typically trained offline and deployed with fixed parameters, their ability to cope with unforeseen changes during operation is limited. Biologically inspired, this work presents a framework for online Continual Reinforcement Learning that enables automated adaptation during deployment. Building on DreamerV3, a model-based Reinforcement Learning algorithm, the proposed method leverages world model prediction residuals to detect out-of-distribution events and automatically trigger finetuning. Adaptation progress is monitored using both task-level performance signals and internal training metrics, allowing convergence to be assessed without external supervision and domain knowledge. The approach is validated on a variety of contemporary continuous control problems, including a quadruped robot in high-fidelity simulation, and a real-world model vehicle. Relevant metrics and their interpretation are presented and discussed, as well as resulting trade-offs described. The results sketch out how autonomous robotic agents could once move beyond static training regimes toward adaptive systems capable of self-reflection and -improvement during operation, just like their biological counterparts.