mlsanityData to 7 Oct 2026

A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning

arXiv 2405.19153 ↗First posted 29 May 2024cs.LG cs.AIcited by 26

Not on the map. Why?

Abstract (arXiv)

Continual learning with deep neural networks presents challenges distinct from both the fixed-dataset and convex continual learning regimes. One such challenge is plasticity loss, wherein a neural network trained in an online fashion displays a degraded ability to fit new tasks. This problem has been extensively studied in both supervised learning and off-policy reinforcement learning (RL), where a number of remedies have been proposed. Still, plasticity loss has received less attention in the on-policy deep RL setting. Here we perform an extensive set of experiments examining plasticity loss and a variety of mitigation methods in on-policy deep RL. We demonstrate that plasticity loss is pervasive under domain shift in this regime, and that a number of methods developed to resolve it in other settings fail, sometimes even performing worse than applying no intervention at all. In contrast, we find that a class of ``regenerative'' methods are able to consistently mitigate plasticity loss in a variety of contexts, including in gridworld tasks and more challenging environments like Montezuma's Revenge and ProcGen.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 21 of its 35 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

21 of 35 references.

References · 21

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016Layer Normalizationcited by 5,579
  3. 2018Exploration by Random Network Distillationcited by 966
  4. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  5. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  6. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  7. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  8. 2023Understanding plasticity in neural networkscited by 170
  9. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
  10. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  11. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  12. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 118
9 more
  1. 2018Learning Montezuma's Revenge from a Single Demonstrationcited by 76
  2. 2020Transient Non-Stationarity and Generalisation in Deep Reinforcement Learningcited by 75
  3. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 71
  4. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 45
  5. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  6. 2023Maintaining Plasticity in Deep Continual Learningcited by 42
  7. 2019Ray Interference: a Source of Plateaus in Deep Reinforcement Learningcited by 40
  8. 2022When Does Re-initialization Work?cited by 12
  9. 2022Neuro-Nav: A Library for Neurally-Plausible Reinforcement Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 15; 2026: 10.

26 papers.

Sort:

By area:

  1. 2025CaRL: Learning Scalable Planning Policies with Simple Rewardscited by 34
  2. 2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
  3. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 22
  4. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Modelscited by 17
  5. 2025A Survey of Continual Reinforcement Learningcited by 16
  6. 2025Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learningcited by 14
  7. 2025The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networkscited by 12
  8. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  9. 2025Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learningcited by 11
  10. 2025Activation Function Design Sustains Plasticity in Continual Learningcited by 6
  11. 2025Spectral Collapse Drives Loss of Plasticity in Deep Continual Learningcited by 5
  12. 2025Plasticine: Accelerating Research in Plasticity-Motivated Deep Reinforcement Learningcited by 2
14 more
  1. 2026SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learningcited by 2
  2. 2025Silent Neuron Theory and Plasticity Preservation for Deep Reinforcement Learning in Adaptive Video Streamingcited by 1
  3. 2025Group Policy Gradientcited by 1
  4. 2025Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learningcited by 1
  5. 2026Mode-Dependent Rectification for Stable PPO Trainingcited by 1
  6. 2025Some Optimizers are More Equal: Understanding the Role of Optimizers in Group Fairnesscited by 0
  7. 2026Rethinking Plasticity in Deep Reinforcement Learningcited by 0
  8. 2026Safe Continual Reinforcement Learning in Non-stationary Environmentscited by 0
  9. 2026PRIME: Plasticity Recovery in Multi-Agent Environments for UAV-Assisted Emergency Communication Networkscited by 0
  10. 2026Adjustment Speed as a Safety Constraint for Nonstationary Reinforcement Learningcited by 0
  11. 2026Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learningcited by 0
  12. 2026What Does a ProcGen Generalization Gap Measure? Action Rules, Residual Entropy, and the Missing Random Floorcited by 0
  13. 2026Rank Collapse Is Recoverable, Growing |Q| Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Criticcited by 0
  14. 2026Continual Reinforcement Learning with Neuroevolutioncited by 0