# A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2405.19153 · HTML: https://mlsanity.com/paper/2405.19153/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 21 of its 35 references that are in the map.
- First posted on arXiv 29 May 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [The Primacy Bias in Deep Reinforcement Learning](https://mlsanity.com/paper/2205.07802/) (2022)
- [The Dormant Neuron Phenomenon in Deep Reinforcement Learning](https://mlsanity.com/paper/2302.12902/) (2023)
- [Understanding plasticity in neural networks](https://mlsanity.com/paper/2303.01486/) (2023)
- [What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study](https://mlsanity.com/paper/2006.05990/) (2020)
- [Bigger, Better, Faster: Human-level Atari with human-level efficiency](https://mlsanity.com/paper/2305.19452/) (2023)
- [Understanding and Preventing Capacity Loss in Reinforcement Learning](https://mlsanity.com/paper/2204.09560/) (2022)
- [Loss of Plasticity in Continual Deep Reinforcement Learning](https://mlsanity.com/paper/2303.07507/) (2023)
- [Learning Montezuma's Revenge from a Single Demonstration](https://mlsanity.com/paper/1812.03381/) (2018)
- [Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning](https://mlsanity.com/paper/2006.05826/) (2020)
- [Deep Reinforcement Learning with Plasticity Injection](https://mlsanity.com/paper/2305.15555/) (2023)
- [DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization](https://mlsanity.com/paper/2310.19668/) (2023)
- [Maintaining Plasticity in Deep Continual Learning](https://mlsanity.com/paper/2306.13812/) (2023)
- [PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning](https://mlsanity.com/paper/2306.10711/) (2023)
- [Ray Interference: a Source of Plateaus in Deep Reinforcement Learning](https://mlsanity.com/paper/1904.11455/) (2019)
- [When Does Re-initialization Work?](https://mlsanity.com/paper/2206.10011/) (2022)
- [Neuro-Nav: A Library for Neurally-Plausible Reinforcement Learning](https://mlsanity.com/paper/2206.03312/) (2022)

## Built on by (most cited first, 26 of 26)

- [CaRL: Learning Scalable Planning Policies with Simple Rewards](https://mlsanity.com/paper/2504.17838/) (2025)
- [Plasticity Loss in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2411.04832/) (2024)
- [Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning](https://mlsanity.com/paper/2506.15544/) (2025)
- [A Survey of Continual Reinforcement Learning](https://mlsanity.com/paper/2506.21872/) (2025)
- [The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models](https://mlsanity.com/paper/2510.02230/) (2025)
- [Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learning](https://mlsanity.com/paper/2505.24061/) (2025)
- [The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks](https://mlsanity.com/paper/2506.03404/) (2025)
- [Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents](https://mlsanity.com/paper/2510.13704/) (2025)
- [Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning](https://mlsanity.com/paper/2506.17204/) (2025)
- [Spectral Collapse Drives Loss of Plasticity in Deep Continual Learning](https://mlsanity.com/paper/2509.22335/) (2025)
- [Activation Function Design Sustains Plasticity in Continual Learning](https://mlsanity.com/paper/2509.22562/) (2025)
- [Plasticine: Accelerating Research in Plasticity-Motivated Deep Reinforcement Learning](https://mlsanity.com/paper/2504.17490/) (2025)
- [Group Policy Gradient](https://mlsanity.com/paper/2510.03679/) (2025)
- [SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learning](https://mlsanity.com/paper/2605.04712/) (2026)
- [Silent Neuron Theory and Plasticity Preservation for Deep Reinforcement Learning in Adaptive Video Streaming](https://mlsanity.com/paper/2505.01584/) (2025)
- [Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learning](https://mlsanity.com/paper/2510.12096/) (2025)
- [Mode-Dependent Rectification for Stable PPO Training](https://mlsanity.com/paper/2602.05619/) (2026)
- [Some Optimizers are More Equal: Understanding the Role of Optimizers in Group Fairness](https://mlsanity.com/paper/2504.14882/) (2025)
- [Rethinking Plasticity in Deep Reinforcement Learning](https://mlsanity.com/paper/2603.21173/) (2026)
- [Safe Continual Reinforcement Learning in Non-stationary Environments](https://mlsanity.com/paper/2604.19737/) (2026)
- [PRIME: Plasticity Recovery in Multi-Agent Environments for UAV-Assisted Emergency Communication Networks](https://mlsanity.com/paper/2607.17922/) (2026)
- [Adjustment Speed as a Safety Constraint for Nonstationary Reinforcement Learning](https://mlsanity.com/paper/2607.21646/) (2026)
- [Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learning](https://mlsanity.com/paper/2607.24996/) (2026)
- [What Does a ProcGen Generalization Gap Measure? Action Rules, Residual Entropy, and the Missing Random Floor](https://mlsanity.com/paper/2609.32532/) (2026)
- [Rank Collapse Is Recoverable, Growing |Q| Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Critic](https://mlsanity.com/paper/2609.32819/) (2026)
- [Continual Reinforcement Learning with Neuroevolution](https://mlsanity.com/paper/2610.01583/) (2026)

## Abstract (arXiv)

Continual learning with deep neural networks presents challenges distinct from both the fixed-dataset and convex continual learning regimes. One such challenge is plasticity loss, wherein a neural network trained in an online fashion displays a degraded ability to fit new tasks. This problem has been extensively studied in both supervised learning and off-policy reinforcement learning (RL), where a number of remedies have been proposed. Still, plasticity loss has received less attention in the on-policy deep RL setting. Here we perform an extensive set of experiments examining plasticity loss and a variety of mitigation methods in on-policy deep RL. We demonstrate that plasticity loss is pervasive under domain shift in this regime, and that a number of methods developed to resolve it in other settings fail, sometimes even performing worse than applying no intervention at all. In contrast, we find that a class of ``regenerative'' methods are able to consistently mitigate plasticity loss in a variety of contexts, including in gridworld tasks and more challenging environments like Montezuma's Revenge and ProcGen.