mlsanityData to 7 Oct 2026

Directions of Curvature as an Explanation for Loss of Plasticity

arXiv 2312.00246 ↗First posted 30 Nov 2023cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

Loss of plasticity is a phenomenon in which neural networks lose their ability to learn from new experience. Despite being empirically observed in several problem settings, little is understood about the mechanisms that lead to loss of plasticity. In this paper, we offer a consistent explanation for loss of plasticity: Neural networks lose directions of curvature during training and that loss of plasticity can be attributed to this reduction in curvature. To support such a claim, we provide a systematic investigation of loss of plasticity across continual learning tasks using MNIST, CIFAR-10 and ImageNet. Our findings illustrate that loss of curvature directions coincides with loss of plasticity, while also showing that previous explanations are insufficient to explain loss of plasticity in all settings. Lastly, we show that regularizers which mitigate loss of plasticity also preserve curvature, motivating a simple distributional regularizer that proves to be effective across the problem settings we considered.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 27 of its 48 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

27 of 48 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  4. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  5. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  6. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 515
  7. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  8. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  9. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  10. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  11. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  12. 2019Meta-Learning Representations for Continual Learningcited by 189
15 more
  1. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  2. 2023Understanding plasticity in neural networkscited by 170
  3. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  4. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  5. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 118
  6. 2016Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Unitscited by 86
  7. 2021Continual Backprop: Stochastic Gradient Descent with Persistent Randomnesscited by 79
  8. 2020Transient Non-Stationarity and Generalisation in Deep Reinforcement Learningcited by 75
  9. 2019Stiffness: A New Perspective on Generalization in Neural Networkscited by 46
  10. 2019Emergent properties of the local geometry of neural loss landscapescited by 46
  11. 2021A study on the plasticity of neural networkscited by 44
  12. 2023Maintaining Plasticity in Deep Continual Learningcited by 42
  13. 2019Harnessing Structures for Value-Based Planning and Reinforcement Learningcited by 24
  14. 2022HesScale: Scalable Computation of Hessian Diagonalscited by 8
  15. 2023Utility-based Perturbed Gradient Descent: An Optimizer for Continual Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 11; 2026: 11.

26 papers.

Sort:

By area:

  1. 2024Learning Continually by Spectral Regularizationcited by 34
  2. 2025Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churncited by 22
  3. 2024Neuroplastic Expansion in Deep Reinforcement Learningcited by 17
  4. 2025Optimizers Qualitatively Alter Solutions And We Should Leverage Thiscited by 17
  5. 2024Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RLcited by 14
  6. 2025Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learningcited by 11
  7. 2025On-device edge learning for IoT data streams: a surveycited by 8
  8. 2025Activation Function Design Sustains Plasticity in Continual Learningcited by 6
  9. 2025Bridging Streaming Continual Learning via In-Context Large Tabular Modelscited by 6
  10. 2025Spectral Collapse Drives Loss of Plasticity in Deep Continual Learningcited by 5
  11. 2026Predicting Plasticity in Deep Continual Learning: A Theoretical Perspectivecited by 4
  12. 2024Reset & Distill: A Recipe for Overcoming Negative Transfer in Continual Reinforcement Learningcited by 3
14 more
  1. 2025Sparsity-Driven Plasticity in Multi-Task Reinforcement Learningcited by 2
  2. 2026Do Neural Networks Lose Plasticity in a Gradually Changing World?cited by 2
  3. 2026The Rank and Gradient Lost in Non-stationarity: Sample Weight Decay for Mitigating Plasticity Loss in Reinforcement Learningcited by 2
  4. 2026SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learningcited by 2
  5. 2025Silent Neuron Theory and Plasticity Preservation for Deep Reinforcement Learning in Adaptive Video Streamingcited by 1
  6. 2025A Unified Noise-Curvature View of Loss of Trainabilitycited by 1
  7. 2025Continual Learning with Query-Only Attentioncited by 1
  8. 2026Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learningcited by 1
  9. 2026Rethinking Plasticity in Deep Reinforcement Learningcited by 0
  10. 2026Attribution-Based Neuron Utility for Plasticity Restoration in Deep Networkscited by 0
  11. 2026Agentic Safety is an Epistemic Property, Not a Behavioral Onecited by 0
  12. 2026Sustaining Plasticity via Learnable Wavelet Activations in Continual Learningcited by 0
  13. 2026SingularClip: Preventing Spectral Collapse to Maintain Plasticity in Continual and Reinforcement Learningcited by 0
  14. 2026Rare Gate Disagreements Can Limit Plasticity: When Gradient Flow Mispredicts Finite-Batch SGDcited by 0