mlsanityData to 7 Oct 2026

Loss of Plasticity in Continual Deep Reinforcement Learning

arXiv 2303.07507 ↗First posted 13 Mar 2023cs.LG cs.AIcited by 118

Not on the map. Why?

Abstract (arXiv)

The ability to learn continually is essential in a complex and changing world. In this paper, we characterize the behavior of canonical value-based deep reinforcement learning (RL) approaches under varying degrees of non-stationarity. In particular, we demonstrate that deep RL agents lose their ability to learn good policies when they cycle through a sequence of Atari 2600 games. This phenomenon is alluded to in prior work under various guises -- e.g., loss of plasticity, implicit under-parameterization, primacy bias, and capacity loss. We investigate this phenomenon closely at scale and analyze how the weights, gradients, and activations change over time in several experiments with varying dimensions (e.g., similarity between games, number of games, number of frames per game), with some experiments spanning 50 days and 2 billion environment interactions. Our analysis shows that the activation footprint of the network becomes sparser, contributing to the diminishing gradients. We investigate a remarkably simple mitigation strategy -- Concatenated ReLUs (CReLUs) activation function -- and demonstrate its effectiveness in facilitating continual learning in a changing environment.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). Builds on 21 of its 37 references that are in the map. First posted on arXiv 13 Mar 2023.

Builds on

21 of 37 references.

References · 21

  1. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  2. 2015Prioritized Experience Replaycited by 1,070
  3. 2022A Generalist Agentcited by 802
  4. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  5. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  6. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  7. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  8. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  9. 2017Noisy Networks for Explorationcited by 262
  10. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  11. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  12. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
9 more
  1. 2018Generalization and Regularization in DQNcited by 117
  2. 2016Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Unitscited by 86
  3. 2021Continual Backprop: Stochastic Gradient Descent with Persistent Randomnesscited by 79
  4. 2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learningcited by 47
  5. 2022MuZero with Self-competition for Rate Control in VP9 Video Compressioncited by 27
  6. 2022Controlling Commercial Cooling Systems Using Reinforcement Learningcited by 25
  7. 2022Investigating the Properties of Neural Network Representations in Reinforcement Learningcited by 24
  8. 2022Deep Inventory Managementcited by 22
  9. 2022Probing Transfer in Deep Reinforcement Learning without Task Engineeringcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 32; 2025: 39; 2026: 33.

118 papers.

Sort:

By area:

  1. 2023A Definition of Continual Reinforcement Learningcited by 98
  2. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  3. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  4. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 71
  5. 2023Maintaining Plasticity in Continual Learning via Regenerative Regularizationcited by 63
  6. 2024Normalization and effective learning rates in reinforcement learningcited by 55
  7. 2025FLEX: Continuous Agent Evolution via Forward Learning from Experiencecited by 51
  8. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  9. 2024Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learningcited by 49
  10. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  11. 2023Maintaining Plasticity in Deep Continual Learningcited by 42
  12. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
38 more
  1. 2024Learning Continually by Spectral Regularizationcited by 34
  2. 2024Weight Clipping for Deep Continual and Reinforcement Learningcited by 29
  3. 2023Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stagescited by 27
  4. 2024No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPOcited by 27
  5. 2023Resetting the Optimizer in Deep RL: An Empirical Studycited by 26
  6. 2023Directions of Curvature as an Explanation for Loss of Plasticitycited by 26
  7. 2024A Study of Plasticity Loss in On-Policy Deep Reinforcement Learningcited by 26
  8. 2023Symmetry Induces Structure and Constraint of Learningcited by 25
  9. 2024Streaming Deep Reinforcement Learning Finally Workscited by 24
  10. 2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
  11. 2025Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churncited by 22
  12. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 22
  13. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  14. 2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21
  15. 2024Parseval Regularization for Continual Reinforcement Learningcited by 21
  16. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergencecited by 17
  17. 2024Neuroplastic Expansion in Deep Reinforcement Learningcited by 17
  18. 2024Non-Stationary Learning of Neural Networks with Automatic Soft Parameter Resetcited by 17
  19. 2024Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learningcited by 16
  20. 2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churncited by 16
  21. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  22. 2025A Survey of Continual Reinforcement Learningcited by 16
  23. 2024Can Learned Optimization Make Reinforcement Learning Less Difficult?cited by 14
  24. 2024Plastic Learning with Deep Fourier Featurescited by 14
  25. 2025Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learningcited by 14
  26. 2026Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learningcited by 13
  27. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  28. 2025Plasticity as the Mirror of Empowermentcited by 11
  29. 2025Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learningcited by 11
  30. 2023Harnessing Discrete Representations For Continual Reinforcement Learningcited by 10
  31. 2024Learning Successor Features the Simple Waycited by 10
  32. 2024Position: Lifetime tuning is incompatible with continual reinforcement learningcited by 9
  33. 2023On the Importance of Feature Decorrelation for Unsupervised Representation Learning in Reinforcement Learningcited by 8
  34. 2024Integrating Present and Past in Unsupervised Continual Learningcited by 8
  35. 2024Remove Symmetries to Control Model Expressivity and Improve Optimizationcited by 8
  36. 2024SPIRE: Synergistic Planning, Imitation, and Reinforcement Learning for Long-Horizon Manipulationcited by 8
  37. 2025MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learningcited by 8
  38. 2025Continual Reinforcement Learning by Planning with Online World Modelscited by 8

50 most cited of 118 All 118 (JSON)