mlsanityData to 14 Oct 2026

Continual Reinforcement Learning for Cyber-Physical Systems: Lessons Learned and Open Challenges

arXiv 2511.15652 ↗First posted 19 Nov 2025cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Continual learning (CL) is a branch of machine learning that aims to enable agents to adapt and generalise previously learned abilities so that these can be reapplied to new tasks or environments. This is particularly useful in multi-task settings or in non-stationary environments, where the dynamics can change over time. This is particularly relevant in cyber-physical systems such as autonomous driving. However, despite recent advances in CL, successfully applying it to reinforcement learning (RL) is still an open problem. This paper highlights open challenges in continual RL (CRL) based on experiments in an autonomous driving environment. In this environment, the agent must learn to successfully park in four different scenarios corresponding to parking spaces oriented at varying angles. The agent is successively trained in these four scenarios one after another, representing a CL environment, using Proximal Policy Optimisation (PPO). These experiments exposed a number of open challenges in CRL: finding suitable abstractions of the environment, oversensitivity to hyperparameters, catastrophic forgetting, and efficient use of neural network capacity. Based on these identified challenges, we present open research questions that are important to be addressed for creating robust CRL systems. In addition, the identified challenges call into question the suitability of neural networks for CL. We also identify the need for interdisciplinary research, in particular between computer science and neuroscience.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 9 of its 14 references that are in the map. First posted on arXiv 19 Nov 2025.

Builds on

9 of 14 references.

References · 9

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  4. 2018Progress & Compress: A scalable framework for continual learningcited by 520
  5. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  6. 2020Supermasks in Superpositioncited by 227
  7. 2018Driving Policy Transfer via Modularity and Abstractioncited by 92
  8. 2022Toward Next-Generation Artificial Intelligence: Catalyzing the NeuroAI Revolutioncited by 78
  9. 2017Scalable Recollections for Continual Lifelong Learningcited by 33

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey of Continual Reinforcement Learning
  2. 2025Advancements and Challenges in Continual Reinforcement Learning: A Comprehensive Review
  3. 2025The Cell Must Go On: Agar.io for Continual Reinforcement Learning
  4. 2026Safe Continual Reinforcement Learning in Non-stationary Environments
  5. 2026Position: Deployed Reinforcement Learning should be Continual
  6. 2026Continual-RL for Generalization in Autonomous Racing on the RoboRacer Platform
  7. 2025Ergodic Risk Measures: Towards a Risk-Aware Foundation for Continual Reinforcement Learning
  8. 2025On the Design of Safe Continual RL Methods for Control of Nonlinear Systems