Don't Forget the Critic: Value-Based Data Rehearsal for Multi-Cyclic Continual Reinforcement Learning
On the map: Deep reinforcement learning methods → Deep RL algorithms and representations
Abstract (arXiv)
Data rehearsal has emerged as a leading approach for mitigating catastrophic forgetting in Continual Reinforcement Learning (CRL). However, existing work remains confined to policy gradient frameworks, regularizing only actors due to the performance degradation incurred by critic regularization. This actor-centric approach overlooks the potential of data rehearsal for value function approximation. Moreover, existing evaluations in CRL rarely consider multi-cyclic environments where task sequences repeat, a critical real-world scenario that exacerbates forgetting and plasticity. We investigate data rehearsal for Deep Q-Networks using Q-value regularization in multi-cyclic settings and propose Qreg+NWLU which introduces two simple modifications: (1) continuous data rehearsal that dynamically collects and updates stored Q-values throughout training, and (2) "No-Wait" regularization that applies immediately rather than after the first task. Together, these modifications yield improvements in learning efficiency, forgetting mitigation, and knowledge transfer over Qreg and conventional CRL methods within value function approximation settings.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 25 references that are in the map. First posted on arXiv 21 May 2026.
Builds on
19 of 25 references.
References · 19
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
- 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
- 2018Experience Replay for Continual Learningcited by 1,095
- 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
- 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
- 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
- 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 521
- 2017Variational Continual Learningcited by 339
- 2019On Tiny Episodic Memories in Continual Learningcited by 307
- 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
- 2018Selective Experience Replay for Lifelong Learningcited by 241
7 more
- 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 121
- 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 116
- 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 97
- 2021CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agentscited by 38
- 2022Disentangling Transfer in Continual Reinforcement Learningcited by 33
- 2024Continual Learning in the Presence of Repetitioncited by 10
- 2023Online Distillation with Continual Learning for Cyclic Domain Shiftscited by 6
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026RL Forgets! Towards Continual Policy Optimization
- 2025A Survey of Continual Reinforcement Learning
- 2024Data Augmentation for Continual RL via Adversarial Gradient Episodic Memory
- 2024Parseval Regularization for Continual Reinforcement Learning
- 2026Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learning
- 2026Taming the Implicit: Dual-Channel Risk-Aware Reinforcement Fine-Tuning for Continual Multimodal Post-Training
- 2025Actor-Free Continuous Control via Structurally Maximizable Q-Functions
- 2026Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?