mlsanityData to 13 Oct 2026

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss

arXiv 2502.01342 ↗First posted 3 Feb 2025cs.LG cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Plasticity loss, a critical challenge in neural network training, limits a model's ability to adapt to new tasks or shifts in data distribution. This paper introduces AID (Activation by Interval-wise Dropout), a novel method inspired by Dropout, designed to address plasticity loss. Unlike Dropout, AID generates subnetworks by applying Dropout with different probabilities on each preactivation interval. Theoretical analysis reveals that AID regularizes the network, promoting behavior analogous to that of deep linear networks, which do not suffer from plasticity loss. We validate the effectiveness of AID in maintaining plasticity across various benchmarks, including continual learning tasks on standard image classification datasets such as CIFAR10, CIFAR100, and TinyImageNet. Furthermore, we show that AID enhances reinforcement learning performance in the Arcade Learning Environment benchmark.

Key facts. Cited by 6 papers in the map (data through 13 Oct 2026). Builds on 30 of its 45 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

30 of 45 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,321
  3. 2016Layer Normalizationcited by 5,624
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,753
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,601
  6. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 517
  7. 2022The Primacy Bias in Deep Reinforcement Learningcited by 243
  8. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 173
  9. 2023Understanding plasticity in neural networkscited by 171
  10. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  11. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  12. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 118
18 more
  1. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 85
  2. 2021Continual Backprop: Stochastic Gradient Descent with Persistent Randomnesscited by 80
  3. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 72
  4. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
  5. 2024Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learningcited by 49
  6. 2021A study on the plasticity of neural networkscited by 44
  7. 2023Maintaining Plasticity in Deep Continual Learningcited by 43
  8. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  9. 2024Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networkscited by 41
  10. 2024Weight Clipping for Deep Continual and Reinforcement Learningcited by 29
  11. 2023Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stagescited by 27
  12. 2023Resetting the Optimizer in Deep RL: An Empirical Studycited by 26
  13. 2020Dropout as an Implicit Gating Mechanism For Continual Learningcited by 24
  14. 2021Adaptive Rational Activations to Boost Deep Reinforcement Learningcited by 23
  15. 2024Plastic Learning with Deep Fourier Featurescited by 14
  16. 2022Consistent Dropout for Policy Gradient Reinforcement Learningcited by 9
  17. 2024DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticitycited by 7
  18. 2024Step Out and Seek Around: On Warm-Start Training with Incremental Datacited by 4

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2026FIRE: Frobenius-Isometry Reinitialization for Balancing the Stability-Plasticity Tradeoffcited by 7
  2. 2025Neuroplasticity in Artificial Intelligence -- An Overview and Inspirations on Drop In & Out Learningcited by 3
  3. 2025Plasticine: Accelerating Research in Plasticity-Motivated Deep Reinforcement Learningcited by 2
  4. 2025Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learningcited by 1
  5. 2025Sequencing to Mitigate Catastrophic Forgetting in Continual Learningcited by 0
  6. 2026Sustaining Plasticity via Learnable Wavelet Activations in Continual Learningcited by 0