# Enabling Continual Learning with Differentiable Hebbian Plasticity

arXiv: https://arxiv.org/abs/2006.16558 · HTML: https://mlsanity.com/paper/2006.16558/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jun 2020.

## Built on by (most cited first, 8 of 8)

- [Understanding plasticity in neural networks](https://mlsanity.com/paper/2303.01486/) (2023)
- [Is Bio-Inspired Learning Better than Backprop? Benchmarking Bio Learning vs. Backprop](https://mlsanity.com/paper/2212.04614/) (2022)
- [Stabilizing Elastic Weight Consolidation method in practical ML tasks and using weight importances for neural network pruning](https://mlsanity.com/paper/2109.10021/) (2021)
- [Improving Performance in Continual Learning Tasks using Bio-Inspired Architectures](https://mlsanity.com/paper/2308.04539/) (2023)
- [Hebbian Continual Representation Learning](https://mlsanity.com/paper/2207.04874/) (2022)
- [Learning to Control Rapidly Changing Synaptic Connections: An Alternative Type of Memory in Sequence Processing Artificial Neural Networks](https://mlsanity.com/paper/2211.09440/) (2022)
- [Empirical investigations on WVA structural issues](https://mlsanity.com/paper/2208.05791/) (2022)
- [Correlation of the importances of neural network weights calculated by modern methods of overcoming catastrophic forgetting](https://mlsanity.com/paper/2211.17012/) (2022)

## Abstract (arXiv)

Continual learning is the problem of sequentially learning new tasks or knowledge while protecting previously acquired knowledge. However, catastrophic forgetting poses a grand challenge for neural networks performing such learning process. Thus, neural networks that are deployed in the real world often struggle in scenarios where the data distribution is non-stationary (concept drift), imbalanced, or not always fully available, i.e., rare edge cases. We propose a Differentiable Hebbian Consolidation model which is composed of a Differentiable Hebbian Plasticity (DHP) Softmax layer that adds a rapid learning plastic component (compressed episodic memory) to the fixed (slow changing) parameters of the softmax output layer; enabling learned representations to be retained for a longer timescale. We demonstrate the flexibility of our method by integrating well-known task-specific synaptic consolidation methods to penalize changes in the slow weights that are important for each target task. We evaluate our approach on the Permuted MNIST, Split MNIST and Vision Datasets Mixture benchmarks, and introduce an imbalanced variant of Permuted MNIST -- a dataset that combines the challenges of class imbalance and concept drift. Our proposed model requires no additional hyperparameters and outperforms comparable baselines by reducing forgetting.