mlsanityData to 14 Oct 2026

Towards Continual Reinforcement Learning: A Review and Perspectives

arXiv 2012.13490 ↗First posted 25 Dec 2020cs.LG cs.AIcited by 274

Not on the map. Why?

Cited by 81 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this article, we aim to provide a literature review of different formulations and approaches to continual reinforcement learning (RL), also known as lifelong or non-stationary RL. We begin by discussing our perspective on why RL is a natural fit for studying continual learning. We then provide a taxonomy of different continual RL formulations by mathematically characterizing two key properties of non-stationarity, namely, the scope and driver non-stationarity. This offers a unified view of various formulations. Next, we review and present a taxonomy of continual RL approaches. We go on to discuss evaluation of continual RL agents, providing an overview of benchmarks used in the literature and important metrics for understanding agent performance. Finally, we highlight open problems and challenges in bridging the gap between the current state of continual RL and findings in neuroscience. While still in its early days, the study of continual RL has the promise to develop better incremental reinforcement learners that can function in increasingly realistic applications where non-stationarity plays a vital role. These include applications such as those in the fields of healthcare, education, logistics, and robotics.

Key facts. Cited by 274 papers in the map (data through 14 Oct 2026). First posted on arXiv 25 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 26; 2022: 41; 2023: 56; 2024: 48; 2025: 51; 2026: 52.

274 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  2. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 176
  3. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  4. 2021Avalanche: an End-to-End Library for Continual Learningcited by 124
  5. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 121
  6. 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 116
  7. 2023A Definition of Continual Reinforcement Learningcited by 101
  8. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  9. 2021Continual Backprop: Stochastic Gradient Descent with Persistent Randomnesscited by 80
  10. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 69
  11. 2022In Defense of the Unitary Scalarization for Deep Multi-Task Learningcited by 67
  12. 2021Continual Learning via Local Module Compositioncited by 63
38 more
  1. 2021A Survey of Deep RL and IL for Autonomous Driving Policy Learningcited by 61
  2. 2024FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuningcited by 54
  3. 2021Universal Off-Policy Evaluationcited by 43
  4. 2021Same State, Different Task: Continual Reinforcement Learning without Interferencecited by 43
  5. 2022A Survey of Ad Hoc Teamwork Researchcited by 43
  6. 2023Improving Language Plasticity via Pretraining with Active Forgettingcited by 43
  7. 2021Visual Adversarial Imitation Learning using Variational Modelscited by 42
  8. 2021CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agentscited by 38
  9. 2023Continual Learning as Computationally Constrained Reinforcement Learningcited by 37
  10. 2021Continuum: Simple Management of Complex Continual Learning Scenarioscited by 33
  11. 2021Autonomous Reinforcement Learning via Subgoal Curriculacited by 33
  12. 2021Autonomous Reinforcement Learning: Formalism and Benchmarkingcited by 33
  13. 2022Disentangling Transfer in Continual Reinforcement Learningcited by 33
  14. 2021Continuous Coordination As a Realistic Scenario for Lifelong Learningcited by 32
  15. 2023Deep Laplacian-based Options for Temporally-Extended Explorationcited by 29
  16. 2022Building a Subspace of Policies for Scalable Continual Learningcited by 28
  17. 2023Learning to Modulate pre-trained Models in RLcited by 28
  18. 2021A Dataset Perspective on Offline Reinforcement Learningcited by 27
  19. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  20. 2023Resetting the Optimizer in Deep RL: An Empirical Studycited by 26
  21. 2021Block Contextual MDPs for Continual Learningcited by 25
  22. 2022You Only Live Once: Single-Life Reinforcement Learningcited by 24
  23. 2023Low-rank extended Kalman filtering for online learning of neural networks from streaming datacited by 23
  24. 2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
  25. 2023Causal Reinforcement Learning: A Surveycited by 23
  26. 2022ConvLab-3: A Flexible Dialogue System Toolkit Based on a Unified Data Formatcited by 22
  27. 2022Lifelong Reinforcement Learning with Modulating Maskscited by 22
  28. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  29. 2024Parseval Regularization for Continual Reinforcement Learningcited by 21
  30. 2021Sequoia: A Software Framework to Unify Continual Learning Researchcited by 19
  31. 2023Prediction and Control in Continual Reinforcement Learningcited by 19
  32. 2022The Alberta Plan for AI Researchcited by 18
  33. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  34. 2022NEVIS'22: A Stream of 100 Tasks Sampled from 30 Years of Computer Vision Researchcited by 18
  35. 2025A Survey of Continual Reinforcement Learningcited by 18
  36. 2022The Effectiveness of World Models for Continual Reinforcement Learningcited by 17
  37. 2025The Future of Continual Learning in the Era of Foundation Models: Three Key Directionscited by 17
  38. 2026Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learningcited by 17

50 most cited of 274 All 274 (JSON)