mlsanityData to 7 Oct 2026

Understanding and Preventing Capacity Loss in Reinforcement Learning

arXiv 2204.09560 ↗First posted 20 Apr 2022cs.LGcited by 124

Not on the map. Why?

Abstract (arXiv)

The reinforcement learning (RL) problem is rife with sources of non-stationarity, making it a notoriously difficult problem domain for the application of neural networks. We identify a mechanism by which non-stationary prediction targets can prevent learning progress in deep RL agents: textitcapacity loss, whereby networks trained on a sequence of target values lose their ability to quickly update their predictions over time. We demonstrate that capacity loss occurs in a range of RL agents and environments, and is particularly damaging to performance in sparse-reward tasks. We then present a simple regularizer, Initial Feature Regularization (InFeR), that mitigates this phenomenon by regressing a subspace of features towards its value at initialization, leading to significant performance improvements in sparse-reward environments such as Montezuma's Revenge. We conclude that preventing capacity loss is crucial to enable agents to maximally benefit from the learning signals they obtain throughout the entire training trajectory.

Key facts. Cited by 124 papers in the map (data through 7 Oct 2026). Builds on 31 of its 48 references that are in the map. First posted on arXiv 20 Apr 2022.

Builds on

31 of 48 references.

References · 31

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2016Learning without Forgettingcited by 2,465
  4. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  6. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  7. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  8. 2017Deep Reinforcement Learning that Matterscited by 727
  9. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  10. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  11. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  12. 2015Policy Distillationcited by 414
19 more
  1. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  2. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  3. 2017DeepStack: Expert-Level Artificial Intelligence in No-Limit Pokercited by 228
  4. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  5. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 171
  6. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  7. 2018Measuring and regularizing networks in function spacecited by 97
  8. 2017Eigenoption Discovery through the Deep Successor Representationcited by 85
  9. 2020Transient Non-Stationarity and Generalisation in Deep Reinforcement Learningcited by 75
  10. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
  11. 2020The Value-Improvement Path: Towards Better Representations for Reinforcement Learningcited by 60
  12. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 56
  13. 2019A Geometric Perspective on Optimal Representations for Reinforcement Learningcited by 54
  14. 2019Discovery of Useful Questions as Auxiliary Taskscited by 45
  15. 2019Ray Interference: a Source of Plateaus in Deep Reinforcement Learningcited by 40
  16. 2021Effective Evaluation of Deep Active Learning on Image Classification Taskscited by 37
  17. 2020Representations for Stable Off-Policy Reinforcement Learningcited by 31
  18. 2020On Catastrophic Interference in Atari 2600 Gamescited by 13
  19. 2021Correcting Momentum in Temporal Difference Learningcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 27; 2024: 41; 2025: 23; 2026: 26.

124 papers.

Sort:

By area:

  1. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  2. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  3. 2023Understanding plasticity in neural networkscited by 170
  4. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  5. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 118
  6. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  7. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 79
  8. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
  9. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
  10. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 71
  11. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 63
  12. 2023Maintaining Plasticity in Continual Learning via Regenerative Regularizationcited by 63
38 more
  1. 2024Normalization and effective learning rates in reinforcement learningcited by 55
  2. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  3. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49
  4. 2023TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Modelscited by 48
  5. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 45
  6. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  7. 2023Maintaining Plasticity in Deep Continual Learningcited by 42
  8. 2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 41
  9. 20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilitiescited by 39
  10. 2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 38
  11. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
  12. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 35
  13. 2024Learning Continually by Spectral Regularizationcited by 34
  14. 2024Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biasescited by 29
  15. 2024Weight Clipping for Deep Continual and Reinforcement Learningcited by 29
  16. 2023Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stagescited by 27
  17. 2024No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPOcited by 27
  18. 2023Resetting the Optimizer in Deep RL: An Empirical Studycited by 26
  19. 2023Directions of Curvature as an Explanation for Loss of Plasticitycited by 26
  20. 2024A Study of Plasticity Loss in On-Policy Deep Reinforcement Learningcited by 26
  21. 2024Streaming Deep Reinforcement Learning Finally Workscited by 24
  22. 2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
  23. 2025Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churncited by 22
  24. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 22
  25. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  26. 2024Parseval Regularization for Continual Reinforcement Learningcited by 21
  27. 2023Prediction and Control in Continual Reinforcement Learningcited by 19
  28. 2024Discovering Temporally-Aware Reinforcement Learning Algorithmscited by 18
  29. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergencecited by 17
  30. 2024Neuroplastic Expansion in Deep Reinforcement Learningcited by 17
  31. 2025Optimizers Qualitatively Alter Solutions And We Should Leverage Thiscited by 17
  32. 2022Adversarial Cheap Talkcited by 16
  33. 2024Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learningcited by 16
  34. 2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churncited by 16
  35. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  36. 2025Learning to Sample Effective and Diverse Prompts for Text-to-Image Generationcited by 16
  37. 2024Can Learned Optimization Make Reinforcement Learning Less Difficult?cited by 14
  38. 2024Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RLcited by 14

50 most cited of 124 All 124 (JSON)