# Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization

arXiv: https://arxiv.org/abs/1802.01569 · HTML: https://mlsanity.com/paper/1802.01569/

## Key facts

- Cited by 104 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Feb 2018.

## Built on by (most cited first, 30 of 104)

- [Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models](https://mlsanity.com/paper/2305.12827/) (2023)
- [Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks](https://mlsanity.com/paper/2112.10017/) (2021)
- [Continual Learning and Catastrophic Forgetting](https://mlsanity.com/paper/2403.05175/) (2024)
- [Class-Incremental Learning with Generative Classifiers](https://mlsanity.com/paper/2104.10093/) (2021)
- [On the Soft-Subnetwork for Few-shot Class Incremental Learning](https://mlsanity.com/paper/2209.07529/) (2022)
- [Continual Learning for Blind Image Quality Assessment](https://mlsanity.com/paper/2102.09717/) (2021)
- [Balancing Stability and Plasticity through Advanced Null Space in Continual Learning](https://mlsanity.com/paper/2207.12061/) (2022)
- [Kernel Continual Learning](https://mlsanity.com/paper/2107.05757/) (2021)
- [Optimizing Reusable Knowledge for Continual Learning via Metalearning](https://mlsanity.com/paper/2106.05390/) (2021)
- [Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments](https://mlsanity.com/paper/2201.00042/) (2021)
- [Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy](https://mlsanity.com/paper/2312.10549/) (2023)
- [DoubleAdapt: A Meta-learning Approach to Incremental Learning for Stock Trend Forecasting](https://mlsanity.com/paper/2306.09862/) (2023)
- [Continual Learning of Diffusion Models with Generative Distillation](https://mlsanity.com/paper/2311.14028/) (2023)
- [Sparsity and Heterogeneous Dropout for Continual Learning in the Null Space of Neural Activations](https://mlsanity.com/paper/2203.06514/) (2022)
- [Disentangling and Mitigating the Impact of Task Similarity for Continual Learning](https://mlsanity.com/paper/2405.20236/) (2024)
- [Long Sequence Hopfield Memory](https://mlsanity.com/paper/2306.04532/) (2023)
- [Two Complementary Perspectives to Continual Learning: Ask Not Only What to Optimize, But Also How](https://mlsanity.com/paper/2311.04898/) (2023)
- [iKAN: Global Incremental Learning with KAN for Human Activity Recognition Across Heterogeneous Datasets](https://mlsanity.com/paper/2406.01646/) (2024)
- [A contrastive rule for meta-learning](https://mlsanity.com/paper/2104.01677/) (2021)
- [Continual Learning via Inter-Task Synaptic Mapping](https://mlsanity.com/paper/2106.13954/) (2021)
- [Persistent Backdoor Attacks in Continual Learning](https://mlsanity.com/paper/2409.13864/) (2024)
- [Do Not Forget to Attend to Uncertainty while Mitigating Catastrophic Forgetting](https://mlsanity.com/paper/2102.01906/) (2021)
- [Modelling continual learning in humans with Hebbian context gating and exponentially decaying task signals](https://mlsanity.com/paper/2203.11560/) (2022)
- [On the Limitations of Continual Learning for Malware Classification](https://mlsanity.com/paper/2208.06568/) (2022)
- [Continual task learning in natural and artificial agents](https://mlsanity.com/paper/2210.04520/) (2022)
- [Continual Learning with Guarantees via Weight Interval Constraints](https://mlsanity.com/paper/2206.07996/) (2022)
- [CP-Prompt: Composition-Based Cross-modal Prompting for Domain-Incremental Continual Learning](https://mlsanity.com/paper/2407.21043/) (2024)
- [Class-Incremental Learning via Knowledge Amalgamation](https://mlsanity.com/paper/2209.02112/) (2022)
- [Statistical mechanics of continual learning: variational principle and mean-field potential](https://mlsanity.com/paper/2212.02846/) (2022)
- [Design Principles for Lifelong Learning AI Accelerators](https://mlsanity.com/paper/2310.04467/) (2023)

## Abstract (arXiv)

Humans and most animals can learn new tasks without forgetting old ones. However, training artificial neural networks (ANNs) on new tasks typically cause it to forget previously learned tasks. This phenomenon is the result of "catastrophic forgetting", in which training an ANN disrupts connection weights that were important for solving previous tasks, degrading task performance. Several recent studies have proposed methods to stabilize connection weights of ANNs that are deemed most important for solving a task, which helps alleviate catastrophic forgetting. Here, drawing inspiration from algorithms that are believed to be implemented in vivo, we propose a complementary method: adding a context-dependent gating signal, such that only sparse, mostly non-overlapping patterns of units are active for any one task. This method is easy to implement, requires little computational overhead, and allows ANNs to maintain high performance across large numbers of sequentially presented tasks when combined with weight stabilization. This work provides another example of how neuroscience-inspired algorithms can benefit ANN design and capability.