# Dropout as an Implicit Gating Mechanism For Continual Learning

arXiv: https://arxiv.org/abs/2004.11545 · HTML: https://mlsanity.com/paper/2004.11545/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2020.

## Built on by (most cited first, 24 of 24)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Wide Neural Networks Forget Less Catastrophically](https://mlsanity.com/paper/2110.11526/) (2021)
- [DualNet: Continual Learning, Fast and Slow](https://mlsanity.com/paper/2110.00175/) (2021)
- [TiC-CLIP: Continual Training of CLIP Models](https://mlsanity.com/paper/2310.16226/) (2023)
- [Balancing Stability and Plasticity through Advanced Null Space in Continual Learning](https://mlsanity.com/paper/2207.12061/) (2022)
- [Implicit regularization of dropout](https://mlsanity.com/paper/2207.05952/) (2022)
- [Sparsity and Heterogeneous Dropout for Continual Learning in the Null Space of Neural Activations](https://mlsanity.com/paper/2203.06514/) (2022)
- [Momentum-based Weight Interpolation of Strong Zero-Shot Models for Continual Learning](https://mlsanity.com/paper/2211.03186/) (2022)
- [Continual Learning Beyond a Single Model](https://mlsanity.com/paper/2202.09826/) (2022)
- [Task-agnostic Continual Learning with Hybrid Probabilistic Models](https://mlsanity.com/paper/2106.12772/) (2021)
- [Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgetting](https://mlsanity.com/paper/2304.00933/) (2023)
- [TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining](https://mlsanity.com/paper/2504.02107/) (2025)
- [Order parameters and phase transitions of continual learning in deep neural networks](https://mlsanity.com/paper/2407.10315/) (2024)
- [Addressing catastrophic forgetting for medical domain expansion](https://mlsanity.com/paper/2103.13511/) (2021)
- [Continual Task Allocation in Meta-Policy Network via Sparse Prompting](https://mlsanity.com/paper/2305.18444/) (2023)
- [Elephant Neural Networks: Born to Be a Continual Learner](https://mlsanity.com/paper/2310.01365/) (2023)
- [STAR: Stability-Inducing Weight Perturbation for Continual Learning](https://mlsanity.com/paper/2503.01595/) (2025)
- [Continual Feature Selection: Spurious Features in Continual Learning](https://mlsanity.com/paper/2203.01012/) (2022)
- [Reset It and Forget It: Relearning Last-Layer Weights Improves Continual and Transfer Learning](https://mlsanity.com/paper/2310.07996/) (2023)
- [Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss](https://mlsanity.com/paper/2502.01342/) (2025)
- [Mitigating Task-Order Sensitivity and Forgetting via Hierarchical Second-Order Consolidation](https://mlsanity.com/paper/2602.02568/) (2026)
- [Design of Explainability Module with Experts in the Loop for Visualization and Dynamic Adjustment of Continual Learning](https://mlsanity.com/paper/2202.06781/) (2022)
- [Continual Learning, Fast and Slow](https://mlsanity.com/paper/2209.02370/) (2022)
- [Efficient Reinforcement Learning by Reducing Forgetting with Elephant Activation Functions](https://mlsanity.com/paper/2509.19159/) (2025)

## Abstract (arXiv)

In recent years, neural networks have demonstrated an outstanding ability to achieve complex learning tasks across various domains. However, they suffer from the "catastrophic forgetting" problem when they face a sequence of learning tasks, where they forget the old ones as they learn new tasks. This problem is also highly related to the "stability-plasticity dilemma". The more plastic the network, the easier it can learn new tasks, but the faster it also forgets previous ones. Conversely, a stable network cannot learn new tasks as fast as a very plastic network. However, it is more reliable to preserve the knowledge it has learned from the previous tasks. Several solutions have been proposed to overcome the forgetting problem by making the neural network parameters more stable, and some of them have mentioned the significance of dropout in continual learning. However, their relationship has not been sufficiently studied yet. In this paper, we investigate this relationship and show that a stable network with dropout learns a gating mechanism such that for different tasks, different paths of the network are active. Our experiments show that the stability achieved by this implicit gating plays a very critical role in leading to performance comparable to or better than other involved continual learning algorithms to overcome catastrophic forgetting.