mlsanityData to 14 Oct 2026

Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn

arXiv 2506.00592 ↗First posted 31 May 2025cs.LG cs.AIcited by 22

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Plasticity, or the ability of an agent to adapt to new tasks, environments, or distributions, is crucial for continual learning. In this paper, we study the loss of plasticity in deep continual RL from the lens of churn: network output variability for out-of-batch data induced by mini-batch training. We demonstrate that (1) the loss of plasticity is accompanied by the exacerbation of churn due to the gradual rank decrease of the Neural Tangent Kernel (NTK) matrix; (2) reducing churn helps prevent rank collapse and adjusts the step size of regular RL gradients adaptively. Moreover, we introduce Continual Churn Approximated Reduction (C-CHAIN) and demonstrate it improves learning performance and outperforms baselines in a diverse range of continual learning environments on OpenAI Gym Control, ProcGen, DeepMind Control Suite, and MinAtar benchmarks.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 54 of its 69 references that are in the map. First posted on arXiv 31 May 2025.

Builds on

54 of 69 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timesteps22 Dec
  2. 2024Parseval Regularization for Continual Reinforcement Learning10 Dec
  3. 2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn7 Sep
  4. 2024Weight Clipping for Deep Continual and Reinforcement Learning1 Jul
  5. 2024Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learning26 May
  6. 2024Disentangling the Causes of Plasticity Loss in Neural Networks29 Feb
  7. 2024In value-based deep reinforcement learning, a pruned network is a good network19 Feb

Other references · 47

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  5. 2016OpenAI Gymcited by 1,736
  6. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  7. 2018Experience Replay for Continual Learningcited by 1,095
  8. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  9. 2017A Closer Look at Memorization in Deep Networkscited by 942
  10. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  11. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  12. 2018Progress & Compress: A scalable framework for continual learningcited by 520
35 more
  1. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 351
  2. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  3. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 204
  4. 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
  5. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  6. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  7. 2024Continual Learning for Large Language Models: A Surveycited by 147
  8. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  9. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
  10. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  11. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 121
  12. 2023A Definition of Continual Reinforcement Learningcited by 101
  13. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 74
  14. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  15. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 65
  16. 2023Maintaining Plasticity in Continual Learning via Regenerative Regularizationcited by 65
  17. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  18. 2024Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learningcited by 50
  19. 2019Towards Characterizing Divergence in Deep Q-Learningcited by 43
  20. 2020Deep Reinforcement Learning amidst Lifelong Non-Stationaritycited by 43
  21. 2023Maintaining Plasticity in Deep Continual Learningcited by 43
  22. 2020Interference and Generalization in Temporal Difference Learningcited by 42
  23. 2023Continual Learning as Computationally Constrained Reinforcement Learningcited by 37
  24. 2023Small batch deep reinforcement learningcited by 36
  25. 2022The Phenomenon of Policy Churncited by 28
  26. 2023Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stagescited by 28
  27. 2023Resetting the Optimizer in Deep RL: An Empirical Studycited by 26
  28. 2023Directions of Curvature as an Explanation for Loss of Plasticitycited by 26
  29. 2019Harnessing Structures for Value-Based Planning and Reinforcement Learningcited by 24
  30. 2019MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experimentscited by 19
  31. 2023Sharing Lifelong Reinforcement Learning Knowledge via Modulating Maskscited by 10
  32. 2023Reset It and Forget It: Relearning Last-Layer Weights Improves Continual and Transfer Learningcited by 6
  33. 2023Measuring and Mitigating Interference in Reinforcement Learningcited by 5
  34. 2024Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learningcited by 5
  35. 2023Tackling Non-Stationarity in Reinforcement Learning via Causal-Origin Representationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 18.

22 papers.

Sort:

By area:

  1. 2025A Survey of Continual Reinforcement Learningcited by 18
  2. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Modelscited by 18
  3. 2026Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learningcited by 17
  4. 2026Sharpness-Aware Pretraining Mitigates Catastrophic Forgettingcited by 10
  5. 2025Spectral Collapse Drives Loss of Plasticity in Deep Continual Learningcited by 6
  6. 2026Stable Deep Reinforcement Learning via Isotropic Gaussian Representationscited by 5
  7. 2026Predicting Plasticity in Deep Continual Learning: A Theoretical Perspectivecited by 4
  8. 2026Do Neural Networks Lose Plasticity in a Gradually Changing World?cited by 2
  9. 2026SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learningcited by 2
  10. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 1
  11. 2026Topology-Aware Revival for Efficient Sparse Trainingcited by 0
  12. 2026A Thermodynamic Theory of Learning Part II: History-Dependent Reachability and Continual Learningcited by 0
10 more
  1. 2026Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learningcited by 0
  2. 2026Vision Transformers that Never Stop Learningcited by 0
  3. 2026Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitionscited by 0
  4. 2026Continual Quadruped Robots Coordination via Semantic Skill Discoverycited by 0
  5. 2026Loss Smoothing for Stable Adaptation Under Distribution Shiftcited by 0
  6. 2026Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learningcited by 0
  7. 2026SingularClip: Preventing Spectral Collapse to Maintain Plasticity in Continual and Reinforcement Learningcited by 0
  8. 2026WarpSAC: Towards the Pinnacle of Scalable Off-policy RL by Rethinking Exploration and Exploitationcited by 0
  9. 2026Rank Collapse Is Recoverable, Growing |Q| Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Criticcited by 0
  10. 2026Continual Reinforcement Learning with Neuroevolutioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning
  2. 2026The Rank and Gradient Lost in Non-stationarity: Sample Weight Decay for Mitigating Plasticity Loss in Reinforcement Learning
  3. 2024Plasticity Loss in Deep Reinforcement Learning: A Survey
  4. 2026Rethinking Plasticity in Deep Reinforcement Learning
  5. 2025Activation Function Design Sustains Plasticity in Continual Learning
  6. 2025Plasticine: Accelerating Research in Plasticity-Motivated Deep Reinforcement Learning
  7. 2026Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning
  8. 2024Temporal-Difference Variational Continual Learning