mlsanityData to 7 Oct 2026

Weight Clipping for Deep Continual and Reinforcement Learning

arXiv 2407.01704 ↗First posted 1 Jul 2024cs.LG cs.AIcited by 29

Not on the map. Why?

Abstract (arXiv)

Many failures in deep continual and reinforcement learning are associated with increasing magnitudes of the weights, making them hard to change and potentially causing overfitting. While many methods address these learning failures, they often change the optimizer or the architecture, a complexity that hinders widespread adoption in various systems. In this paper, we focus on learning failures that are associated with increasing weight norm and we propose a simple technique that can be easily added on top of existing learning systems: clipping neural network weights to limit them to a specific range. We study the effectiveness of weight clipping in a series of supervised and reinforcement learning experiments. Our empirical results highlight the benefits of weight clipping for generalization, addressing loss of plasticity and policy collapse, and facilitating learning with a large replay ratio.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 38 of its 54 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

38 of 54 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2023Mastering Diverse Domains through World Modelscited by 1,193
  7. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  8. 2018On Lazy Training in Differentiable Programmingcited by 671
  9. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  10. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  11. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
  12. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
26 more
  1. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  2. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  3. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 178
  4. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  5. 2023Understanding plasticity in neural networkscited by 170
  6. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 139
  7. 2018Memory Efficient Experience Replay for Streaming Learningcited by 128
  8. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  9. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 118
  10. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  11. 2019Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networkscited by 86
  12. 2022Learning Smooth Neural Functions via Lipschitz Regularizationcited by 86
  13. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  14. 2021Continual Backprop: Stochastic Gradient Descent with Persistent Randomnesscited by 79
  15. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 71
  16. 2024Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learningcited by 49
  17. 2022Online Continual Learning for Embedded Devicescited by 48
  18. 2019Stiffness: A New Perspective on Generalization in Neural Networkscited by 46
  19. 2023Continual Learning as Computationally Constrained Reinforcement Learningcited by 35
  20. 2020Learning compositional functions via multiplicative weight updatescited by 20
  21. 2022Learning Dynamics and Generalization in Reinforcement Learningcited by 11
  22. 2023Elephant Neural Networks: Born to Be a Continual Learnercited by 10
  23. 2021An Alternate Policy Gradient Estimator for Softmax Policiescited by 9
  24. 2022Memory-efficient Reinforcement Learning with Value-based Knowledge Consolidationcited by 9
  25. 2016Synaptic plasticity and neuronal refractory time cause scaling behaviour of neuronal avalanchescited by 1
  26. 2023DP-SGD with weight clippingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 14; 2026: 11.

29 papers.

Sort:

  1. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 81
  2. 2024Streaming Deep Reinforcement Learning Finally Workscited by 24
  3. 2025Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churncited by 22
  4. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 22
  5. 2025Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalizationcited by 19
  6. 2024Neuroplastic Expansion in Deep Reinforcement Learningcited by 17
  7. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  8. 2024Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Bufferscited by 16
  9. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  10. 2025What Can Grokking Teach Us About Learning Under Nonstationarity?cited by 7
  11. 2026FIRE: Frobenius-Isometry Reinitialization for Balancing the Stability-Plasticity Tradeoffcited by 7
  12. 2025Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Losscited by 6
17 more
  1. 2025Reinitializing weights vs units for maintaining plasticity in neural networkscited by 6
  2. 2025Weight-Space Linear Recurrent Neural Networkscited by 5
  3. 2026FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Controlcited by 5
  4. 2025Variance Control via Weight Rescaling in LLM Pre-trainingcited by 4
  5. 2025Silent Neuron Theory and Plasticity Preservation for Deep Reinforcement Learning in Adaptive Video Streamingcited by 1
  6. 2025Recovering Plasticity of Neural Networks via Soft Weight Rescalingcited by 1
  7. 2025Dynamic Object Masks as Goal Representations for Visual Goal-Conditioned Reinforcement Learningcited by 1
  8. 2026Learning to Forget: Continual Learning with Adaptive Weight Decaycited by 1
  9. 2026Angel or Demon: Investigating the Plasticity Interventions' Impact on Backdoor Threats in Deep Reinforcement Learningcited by 1
  10. 2026Can Scale Save Us From Plasticity Loss in Large Language Models?cited by 1
  11. 2025Iterated Population Based Training with Task-Agnostic Restartscited by 0
  12. 2026Rethinking Plasticity in Deep Reinforcement Learningcited by 0
  13. 2026Rotation-Preserving Supervised Fine-Tuningcited by 0
  14. 2026When is Warmstarting Effective for Scaling Language Models?cited by 0
  15. 2026Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitionscited by 0
  16. 2026PRIME: Plasticity Recovery in Multi-Agent Environments for UAV-Assisted Emergency Communication Networkscited by 0
  17. 2026Sustaining Plasticity via Learnable Wavelet Activations in Continual Learningcited by 0