# Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn

arXiv: https://arxiv.org/abs/2506.00592 · HTML: https://mlsanity.com/paper/2506.00592/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 54 of its 69 references that are in the map.
- First posted on arXiv 31 May 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 54)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [The Pitfalls of Simplicity Bias in Neural Networks](https://mlsanity.com/paper/2006.07710/) (2020)
- [The Primacy Bias in Deep Reinforcement Learning](https://mlsanity.com/paper/2205.07802/) (2022)
- [SGD on Neural Networks Learns Functions of Increasing Complexity](https://mlsanity.com/paper/1905.11604/) (2019)
- [Deep Reinforcement Learning and the Deadly Triad](https://mlsanity.com/paper/1812.02648/) (2018)
- [The Dormant Neuron Phenomenon in Deep Reinforcement Learning](https://mlsanity.com/paper/2302.12902/) (2023)
- [Bigger, Better, Faster: Human-level Atari with human-level efficiency](https://mlsanity.com/paper/2305.19452/) (2023)
- [Continual Learning for Large Language Models: A Survey](https://mlsanity.com/paper/2402.01364/) (2024)
- [Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning](https://mlsanity.com/paper/2010.14498/) (2020)
- [Understanding and Preventing Capacity Loss in Reinforcement Learning](https://mlsanity.com/paper/2204.09560/) (2022)
- [Label Noise SGD Provably Prefers Flat Global Minimizers](https://mlsanity.com/paper/2106.06530/) (2021)
- [Loss of Plasticity in Continual Deep Reinforcement Learning](https://mlsanity.com/paper/2303.07507/) (2023)
- [A Definition of Continual Reinforcement Learning](https://mlsanity.com/paper/2307.11046/) (2023)
- [Disentangling the Causes of Plasticity Loss in Neural Networks](https://mlsanity.com/paper/2402.18762/) (2024)
- [Deep Reinforcement Learning with Plasticity Injection](https://mlsanity.com/paper/2305.15555/) (2023)
- [DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization](https://mlsanity.com/paper/2112.04716/) (2021)
- [Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective](https://mlsanity.com/paper/2105.05246/) (2021)
- [Maintaining Plasticity in Continual Learning via Regenerative Regularization](https://mlsanity.com/paper/2308.11958/) (2023)
- [Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning](https://mlsanity.com/paper/2403.00514/) (2024)

## Built on by (most cited first, 22 of 22)

- [A Survey of Continual Reinforcement Learning](https://mlsanity.com/paper/2506.21872/) (2025)
- [The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models](https://mlsanity.com/paper/2510.02230/) (2025)
- [Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learning](https://mlsanity.com/paper/2603.11653/) (2026)
- [Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting](https://mlsanity.com/paper/2605.02105/) (2026)
- [Spectral Collapse Drives Loss of Plasticity in Deep Continual Learning](https://mlsanity.com/paper/2509.22335/) (2025)
- [Stable Deep Reinforcement Learning via Isotropic Gaussian Representations](https://mlsanity.com/paper/2602.19373/) (2026)
- [Predicting Plasticity in Deep Continual Learning: A Theoretical Perspective](https://mlsanity.com/paper/2605.09044/) (2026)
- [Do Neural Networks Lose Plasticity in a Gradually Changing World?](https://mlsanity.com/paper/2602.09234/) (2026)
- [SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learning](https://mlsanity.com/paper/2605.04712/) (2026)
- [Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies](https://mlsanity.com/paper/2508.03194/) (2025)
- [Topology-Aware Revival for Efficient Sparse Training](https://mlsanity.com/paper/2602.04166/) (2026)
- [A Thermodynamic Theory of Learning Part II: History-Dependent Reachability and Continual Learning](https://mlsanity.com/paper/2602.07950/) (2026)
- [Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learning](https://mlsanity.com/paper/2603.00903/) (2026)
- [Vision Transformers that Never Stop Learning](https://mlsanity.com/paper/2603.07787/) (2026)
- [Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions](https://mlsanity.com/paper/2606.03382/) (2026)
- [Continual Quadruped Robots Coordination via Semantic Skill Discovery](https://mlsanity.com/paper/2606.08102/) (2026)
- [Loss Smoothing for Stable Adaptation Under Distribution Shift](https://mlsanity.com/paper/2607.00634/) (2026)
- [Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learning](https://mlsanity.com/paper/2607.24996/) (2026)
- [SingularClip: Preventing Spectral Collapse to Maintain Plasticity in Continual and Reinforcement Learning](https://mlsanity.com/paper/2608.18319/) (2026)
- [WarpSAC: Towards the Pinnacle of Scalable Off-policy RL by Rethinking Exploration and Exploitation](https://mlsanity.com/paper/2608.24479/) (2026)
- [Rank Collapse Is Recoverable, Growing |Q| Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Critic](https://mlsanity.com/paper/2609.32819/) (2026)
- [Continual Reinforcement Learning with Neuroevolution](https://mlsanity.com/paper/2610.01583/) (2026)

## Abstract (arXiv)

Plasticity, or the ability of an agent to adapt to new tasks, environments, or distributions, is crucial for continual learning. In this paper, we study the loss of plasticity in deep continual RL from the lens of churn: network output variability for out-of-batch data induced by mini-batch training. We demonstrate that (1) the loss of plasticity is accompanied by the exacerbation of churn due to the gradual rank decrease of the Neural Tangent Kernel (NTK) matrix; (2) reducing churn helps prevent rank collapse and adjusts the step size of regular RL gradients adaptively. Moreover, we introduce Continual Churn Approximated Reduction (C-CHAIN) and demonstrate it improves learning performance and outperforms baselines in a diverse range of continual learning environments on OpenAI Gym Control, ProcGen, DeepMind Control Suite, and MinAtar benchmarks.