# Continual Backprop: Stochastic Gradient Descent with Persistent Randomness

arXiv: https://arxiv.org/abs/2108.06325 · HTML: https://mlsanity.com/paper/2108.06325/

## Key facts

- Cited by 80 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 53 references that are in the map.
- First posted on arXiv 13 Aug 2021.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Online Learning: A Modern Introduction Using Convex Optimization](https://mlsanity.com/paper/1912.13213/) (2019)
- [Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning](https://mlsanity.com/paper/1803.11347/) (2018)
- [Towards Continual Reinforcement Learning: A Review and Perspectives](https://mlsanity.com/paper/2012.13490/) (2020)
- [Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures](https://mlsanity.com/paper/1607.03250/) (2016)
- [Online Learning: A Comprehensive Survey](https://mlsanity.com/paper/1802.02871/) (2018)
- [Meta-Learning Representations for Continual Learning](https://mlsanity.com/paper/1905.12588/) (2019)
- [Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments](https://mlsanity.com/paper/1710.03641/) (2017)
- [Online Deep Learning: Learning Deep Neural Networks on the Fly](https://mlsanity.com/paper/1711.03705/) (2017)
- [Growing a Brain: Fine-Tuning by Increasing Model Capacity](https://mlsanity.com/paper/1907.07844/) (2019)
- [Online Normalization for Training Neural Networks](https://mlsanity.com/paper/1905.05894/) (2019)

## Built on by (most cited first, 30 of 80)

- [The Primacy Bias in Deep Reinforcement Learning](https://mlsanity.com/paper/2205.07802/) (2022)
- [The Dormant Neuron Phenomenon in Deep Reinforcement Learning](https://mlsanity.com/paper/2302.12902/) (2023)
- [Understanding plasticity in neural networks](https://mlsanity.com/paper/2303.01486/) (2023)
- [Loss of Plasticity in Continual Deep Reinforcement Learning](https://mlsanity.com/paper/2303.07507/) (2023)
- [A Definition of Continual Reinforcement Learning](https://mlsanity.com/paper/2307.11046/) (2023)
- [Disentangling the Causes of Plasticity Loss in Neural Networks](https://mlsanity.com/paper/2402.18762/) (2024)
- [Mixtures of Experts Unlock Parameter Scaling for Deep RL](https://mlsanity.com/paper/2402.08609/) (2024)
- [Deep Reinforcement Learning with Plasticity Injection](https://mlsanity.com/paper/2305.15555/) (2023)
- [Maintaining Plasticity in Continual Learning via Regenerative Regularization](https://mlsanity.com/paper/2308.11958/) (2023)
- [Normalization and effective learning rates in reinforcement learning](https://mlsanity.com/paper/2407.01800/) (2024)
- [Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning](https://mlsanity.com/paper/2403.00514/) (2024)
- [Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning](https://mlsanity.com/paper/2404.00781/) (2024)
- [Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks](https://mlsanity.com/paper/2410.23208/) (2024)
- [Meta-Learning Online Adaptation of Language Models](https://mlsanity.com/paper/2305.15076/) (2023)
- [PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning](https://mlsanity.com/paper/2306.10711/) (2023)
- [Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Prediction](https://mlsanity.com/paper/2301.10034/) (2023)
- [Continual Learning as Computationally Constrained Reinforcement Learning](https://mlsanity.com/paper/2307.04345/) (2023)
- [Prototype-Sample Relation Distillation: Towards Replay-Free Continual Learning](https://mlsanity.com/paper/2303.14771/) (2023)
- [Learning Continually by Spectral Regularization](https://mlsanity.com/paper/2406.06811/) (2024)
- [Weight Clipping for Deep Continual and Reinforcement Learning](https://mlsanity.com/paper/2407.01704/) (2024)
- [Training for the Future: A Simple Gradient Interpolation Loss to Generalize Along Time](https://mlsanity.com/paper/2108.06721/) (2021)
- [Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages](https://mlsanity.com/paper/2310.07418/) (2023)
- [No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO](https://mlsanity.com/paper/2405.00662/) (2024)
- [Directions of Curvature as an Explanation for Loss of Plasticity](https://mlsanity.com/paper/2312.00246/) (2023)
- [Low-rank extended Kalman filtering for online learning of neural networks from streaming data](https://mlsanity.com/paper/2305.19535/) (2023)
- [Mixture of Experts in a Mixture of RL settings](https://mlsanity.com/paper/2406.18420/) (2024)
- [Parseval Regularization for Continual Reinforcement Learning](https://mlsanity.com/paper/2412.07224/) (2024)
- [Prediction and Control in Continual Reinforcement Learning](https://mlsanity.com/paper/2312.11669/) (2023)
- [Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learning](https://mlsanity.com/paper/2405.16642/) (2024)
- [Non-Stationary Learning of Neural Networks with Automatic Soft Parameter Reset](https://mlsanity.com/paper/2411.04034/) (2024)

## Abstract (arXiv)

The Backprop algorithm for learning in neural networks utilizes two mechanisms: first, stochastic gradient descent and second, initialization with small random weights, where the latter is essential to the effectiveness of the former. We show that in continual learning setups, Backprop performs well initially, but over time its performance degrades. Stochastic gradient descent alone is insufficient to learn continually; the initial randomness enables only initial learning but not continual learning. To the best of our knowledge, ours is the first result showing this degradation in Backprop's ability to learn. To address this degradation in Backprop's plasticity, we propose an algorithm that continually injects random features alongside gradient descent using a new generate-and-test process. We call this the textitContinual Backprop algorithm. We show that, unlike Backprop, Continual Backprop is able to continually adapt in both supervised and reinforcement learning (RL) problems. Continual Backprop has the same computational complexity as Backprop and can be seen as a natural extension of Backprop for continual learning.