# Position: Deployed Reinforcement Learning should be Continual

arXiv: https://arxiv.org/abs/2606.04029 · HTML: https://mlsanity.com/paper/2606.04029/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 42 of its 88 references that are in the map.
- First posted on arXiv 1 Jun 2026.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 42)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Performative Prediction](https://mlsanity.com/paper/2002.06673/) (2020)
- [Challenges of Real-World Reinforcement Learning](https://mlsanity.com/paper/1904.12901/) (2019)
- [Towards Continual Reinforcement Learning: A Review and Perspectives](https://mlsanity.com/paper/2012.13490/) (2020)
- [Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks](https://mlsanity.com/paper/1812.07252/) (2018)
- [A Survey of Zero-shot Generalisation in Deep Reinforcement Learning](https://mlsanity.com/paper/2111.09794/) (2021)
- [Loss of Plasticity in Continual Deep Reinforcement Learning](https://mlsanity.com/paper/2303.07507/) (2023)
- [Continual World: A Robotic Benchmark For Continual Reinforcement Learning](https://mlsanity.com/paper/2105.10919/) (2021)
- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Safe Reinforcement Learning by Imagining the Near Future](https://mlsanity.com/paper/2202.07789/) (2022)
- [Continual Learning: Applications and the Road Forward](https://mlsanity.com/paper/2311.11908/) (2023)
- [MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Data](https://mlsanity.com/paper/2406.18321/) (2024)
- [Continual Learning as Computationally Constrained Reinforcement Learning](https://mlsanity.com/paper/2307.04345/) (2023)
- [Object Detection Using Sim2Real Domain Randomization for Robotic Applications](https://mlsanity.com/paper/2208.04171/) (2022)
- [Prediction and Control in Continual Reinforcement Learning](https://mlsanity.com/paper/2312.11669/) (2023)
- [The Alberta Plan for AI Research](https://mlsanity.com/paper/2208.11173/) (2022)
- [Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers](https://mlsanity.com/paper/2411.15370/) (2024)
- [Rethinking the Foundations for Continual Reinforcement Learning](https://mlsanity.com/paper/2504.08161/) (2025)
- [Three Dogmas of Reinforcement Learning](https://mlsanity.com/paper/2407.10583/) (2024)
- [Goal-Space Planning with Subgoal Models](https://mlsanity.com/paper/2206.02902/) (2022)
- [Scalable Real-Time Recurrent Learning Using Columnar-Constructive Networks](https://mlsanity.com/paper/2302.05326/) (2023)
- [Position: Lifetime tuning is incompatible with continual reinforcement learning](https://mlsanity.com/paper/2404.02113/) (2024)

## Built on by (most cited first, 1 of 1)

- [Space-sampled Value Decay: Forgetting Mechanisms for Non-stationary Reinforcement Learning](https://mlsanity.com/paper/2606.11797/) (2026)

## Abstract (arXiv)

Reinforcement Learning (RL) has received increasing attention and adoption in real-world use cases. Most of these systems follow a train-then-fix paradigm, where trained agents do not learn while interacting with the world until performance degrades and retraining becomes necessary. In this position paper, we argue that deploying an agent that is incapable of optimality, but receives an evaluative reward signal, is inherently a continual RL problem. We identify four sources of non-stationarity after deployment that necessitate never-ending learning, and highlight why the best deployed agents never stop adapting. We analyze successful examples of continual RL in the real world, and present the community with the advantages and measures to move away from the current train-then-fix paradigm.