# Adaptive Online Planning for Continual Lifelong Learning

arXiv: https://arxiv.org/abs/1912.01188 · HTML: https://mlsanity.com/paper/1912.01188/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Dec 2019.

## Built on by (most cited first, 7 of 7)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Continual World: A Robotic Benchmark For Continual Reinforcement Learning](https://mlsanity.com/paper/2105.10919/) (2021)
- [Prediction and Control in Continual Reinforcement Learning](https://mlsanity.com/paper/2312.11669/) (2023)
- [Reset-free Reinforcement Learning with World Models](https://mlsanity.com/paper/2408.09807/) (2024)
- [Lifelong Reinforcement Learning with Similarity-Driven Weighting by Large Models](https://mlsanity.com/paper/2503.12923/) (2025)
- [Dynamic Entropy Tuning in Reinforcement Learning Low-Level Quadcopter Control: Stochasticity vs Determinism](https://mlsanity.com/paper/2512.18336/) (2025)
- [Neural-Progressive Hedging: Enforcing Constraints in Reinforcement Learning with Stochastic Programming](https://mlsanity.com/paper/2202.13436/) (2022)

## Abstract (arXiv)

We study learning control in an online reset-free lifelong learning scenario, where mistakes can compound catastrophically into the future and the underlying dynamics of the environment may change. Traditional model-free policy learning methods have achieved successes in difficult tasks due to their broad flexibility, but struggle in this setting, as they can activate failure modes early in their lifetimes which are difficult to recover from and face performance degradation as dynamics change. On the other hand, model-based planning methods learn and adapt quickly, but require prohibitive levels of computational resources. We present a new algorithm, Adaptive Online Planning (AOP), that achieves strong performance in this setting by combining model-based planning with model-free learning. By approximating the uncertainty of the model-free components and the planner performance, AOP is able to call upon more extensive planning only when necessary, leading to reduced computation times, while still gracefully adapting behaviors in the face of unpredictable changes in the world -- even when traditional RL fails.