mlsanityData to 14 Oct 2026

A Survey of Continual Reinforcement Learning

arXiv 2506.21872 ↗First posted 27 Jun 2025cs.LG cs.AIcited by 18

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Reinforcement Learning (RL) is an important machine learning paradigm for solving sequential decision-making problems. Recent years have witnessed remarkable progress in this field due to the rapid development of deep neural networks. However, the success of RL currently relies on extensive training data and computational resources. In addition, RL's limited ability to generalize across tasks restricts its applicability in dynamic and real-world environments. With the arisen of Continual Learning (CL), Continual Reinforcement Learning (CRL) has emerged as a promising research direction to address these limitations by enabling agents to learn continuously, adapt to new tasks, and retain previously acquired knowledge. In this survey, we provide a comprehensive examination of CRL, focusing on its core concepts, challenges, and methodologies. Firstly, we conduct a detailed review of existing works, organizing and analyzing their metrics, tasks, benchmarks, and scenario settings. Secondly, we propose a new taxonomy of CRL methods, categorizing them into four types from the perspective of knowledge storage and/or transfer. Finally, our analysis highlights the unique challenges of CRL and provides practical insights into future directions.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 162 of its 246 references that are in the map. First posted on arXiv 27 Jun 2025.

Builds on

162 of 246 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025Self-Composing Policies for Scalable Continual Reinforcement Learning4 Jun
  2. 2025Unsupervised Meta-Testing with Conditional Neural Processes for Hybrid Meta-Reinforcement Learning4 Jun
  3. 2025Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn31 May
  4. 2025Rethinking the Foundations for Continual Reinforcement Learning10 Apr
  5. 2025Knowledge Retention for Continual Model-Based Reinforcement Learning6 Mar
  6. 2025A Survey of In-Context Reinforcement Learning11 Feb
  7. 2024Plasticity Loss in Deep Reinforcement Learning: A Survey7 Nov
  8. 2024Learning Successor Features the Simple Way29 Oct
  9. 2024HackAtari: Atari Learning Environments for Robust and Continual Reinforcement Learning6 Jun
  10. 2024A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning29 May
  11. 2024Statistical Context Detection for Deep Lifelong Reinforcement Learning29 May
  12. 2024Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learning26 May
5 more
  1. 2024PEAC: Unsupervised Pre-training for Cross-Embodiment Reinforcement Learning23 May
  2. 2024Position: Lifetime tuning is incompatible with continual reinforcement learning2 Apr
  3. 2024Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings27 Feb
  4. 2024Multi-granularity Knowledge Transfer for Continual Reinforcement Learning25 Jan
  5. 2024Locality Sensitive Sparse Encoding for Learning World Models Online23 Jan

Other references · 145

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2015Continuous control with deep reinforcement learningcited by 3,407
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2015Trust Region Policy Optimizationcited by 2,580
  10. 2016Learning without Forgettingcited by 2,496
  11. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  12. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
38 more
  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  3. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  4. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  5. 2016Progressive Neural Networkscited by 1,361
  6. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  8. 2017Continual Learning with Deep Generative Replaycited by 1,158
  9. 2015Prioritized Experience Replaycited by 1,099
  10. 2018Experience Replay for Continual Learningcited by 1,095
  11. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  12. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  13. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  14. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  15. 2018Progress & Compress: A scalable framework for continual learningcited by 520
  16. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  17. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  18. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
  19. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 461
  20. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 449
  21. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 364
  22. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  23. 2023Language to Rewards for Robotic Skill Synthesiscited by 312
  24. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 276
  25. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 274
  26. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  27. 2019Continual learning with hypernetworkscited by 273
  28. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 265
  29. 2023Reward Design with Language Modelscited by 261
  30. 2018Selective Experience Replay for Lifelong Learningcited by 241
  31. 2018Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselinescited by 228
  32. 2020Supermasks in Superpositioncited by 227
  33. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 214
  34. 2019The Hanabi Challenge: A New Frontier for AI Researchcited by 197
  35. 2021Few-Shot Incremental Learning with Continually Evolved Classifierscited by 187
  36. 2019Class-incremental Learning via Deep Model Consolidationcited by 173
  37. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 121
  38. 2022Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidationcited by 113

50 most cited of 145 All 145 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 17.

18 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2026Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learningcited by 17
  3. 2026CRL-VLA: Continual Vision-Language-Action Learningcited by 7
  4. 2026Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stagescited by 6
  5. 2026PATH-Bench: Path-Dependent Evaluation of Lifelong Agentscited by 3
  6. 2026Can You Hear, Localize, and Segment Continually? An Exemplar-Free Continual Learning Benchmark for Audio-Visual Segmentationcited by 1
  7. 2026CRoSS: A Continual Robotic Simulation Suite for Scalable Reinforcement Learning with High Task Diversity and Realistic Physics Simulationcited by 0
  8. 2026An LLM-Driven Closed-Loop Autonomous Learning Framework for Robots Facing Uncovered Tasks in Open Environmentscited by 0
  9. 2026Regime-Adaptive Continual Learning for Portfolio Managementcited by 0
  10. 2026Training Prompt Matters: State-Adaptive Optimization for Robust Fine-Tuningcited by 0
  11. 2026Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitionscited by 0
  12. 2026Emergent Language as an Approach to Conscious AIcited by 0
6 more
  1. 2026Continual Quadruped Robots Coordination via Semantic Skill Discoverycited by 0
  2. 2026Offline Multi-agent Continual Cooperation via Skill Partition and Reusecited by 0
  3. 2026Continual-RL for Generalization in Autonomous Racing on the RoboRacer Platformcited by 0
  4. 2026A MARL Centered Reference Architecture for Large Language Model Augmentation in Smart Manufacturingcited by 0
  5. 2026Continual Reinforcement Learning with Neuroevolutioncited by 0
  6. 2026Continual Graph Multi-Agent Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Advancements and Challenges in Continual Reinforcement Learning: A Comprehensive Review
  2. 2025Continual Reinforcement Learning for Cyber-Physical Systems: Lessons Learned and Open Challenges
  3. 2024A Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges
  4. 2025Ergodic Risk Measures: Towards a Risk-Aware Foundation for Continual Reinforcement Learning
  5. 2026Position: Deployed Reinforcement Learning should be Continual
  6. 2023Understanding Reinforcement Learning Algorithms: The Progress from Basic Q-learning to Proximal Policy Optimization
  7. 2025The Cell Must Go On: Agar.io for Continual Reinforcement Learning
  8. 2024Reset & Distill: A Recipe for Overcoming Negative Transfer in Continual Reinforcement Learning