# Stabilizing Elastic Weight Consolidation method in practical ML tasks and using weight importances for neural network pruning

arXiv: https://arxiv.org/abs/2109.10021 · HTML: https://mlsanity.com/paper/2109.10021/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 22 references that are in the map.
- First posted on arXiv 21 Sep 2021.

## Builds on (most cited first, 11 of 11)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [Differentiable plasticity: training plastic neural networks with backpropagation](https://mlsanity.com/paper/1804.02464/) (2018)
- [On Quadratic Penalties in Elastic Weight Consolidation](https://mlsanity.com/paper/1712.03847/) (2017)
- [Addressing catastrophic forgetting for medical domain expansion](https://mlsanity.com/paper/2103.13511/) (2021)
- [Towards continuous learning for glioma segmentation with elastic weight consolidation](https://mlsanity.com/paper/1909.11479/) (2019)
- [Enabling Continual Learning with Differentiable Hebbian Plasticity](https://mlsanity.com/paper/2006.16558/) (2020)
- [Sequential Domain Adaptation through Elastic Weight Consolidation for Sentiment Analysis](https://mlsanity.com/paper/2007.01189/) (2020)
- [Natural Way to Overcome the Catastrophic Forgetting in Neural Networks](https://mlsanity.com/paper/2005.07107/) (2020)

## Built on by (most cited first, 8 of 8)

- [CRL-VLA: Continual Vision-Language-Action Learning](https://mlsanity.com/paper/2602.03445/) (2026)
- [Intelligent Learning Rate Distribution to reduce Catastrophic Forgetting in Transformers](https://mlsanity.com/paper/2404.01317/) (2024)
- [Towards continually learning new languages](https://mlsanity.com/paper/2211.11703/) (2022)
- [Robust Active Learning (RoAL): Countering Dynamic Adversaries in Active Learning with Elastic Weight Consolidation](https://mlsanity.com/paper/2408.07364/) (2024)
- [Empirical investigations on WVA structural issues](https://mlsanity.com/paper/2208.05791/) (2022)
- [Correlation of the importances of neural network weights calculated by modern methods of overcoming catastrophic forgetting](https://mlsanity.com/paper/2211.17012/) (2022)
- [Shift Detection and Adaptation for Network Intrusion Detection](https://mlsanity.com/paper/2508.15100/) (2025)
- [RoAD Benchmark: How LiDAR Models Fail under Coupled Domain Shifts and Label Evolution](https://mlsanity.com/paper/2601.07855/) (2026)

## Abstract (arXiv)

This paper is devoted to the features of the practical application of the Elastic Weight Consolidation (EWC) method for continual learning of neural networks on several training sets. We will more rigorously compare the well-known methodologies for calculating the importance of weights used in the EWC method. These are the Memory Aware Synapses (MAS), Synaptic Intelligence (SI) methodologies and the calculation of the importance of weights based on the Fisher information matrix from the original paper on EWC. We will consider these methodologies as applied to deep neural networks with fully connected and convolutional layers, find the optimal hyperparameters for each of the methodologies, and compare the results of continual neural network learning using these hyperparameters. Next, we will point out the problems that arise when applying the EWC method to deep neural networks with convolutional layers and self-attention layers, such as the "gradient explosion" and the loss of meaningful information in the gradient when using the constraint of its norm (gradient clipping). Then, we will propose a stabilization approach for the EWC method that helps to solve these problems, evaluate it in comparison with the original methodology and show that the proposed stabilization approach performs on the task of maintaining skills during continual learning no worse than the original EWC, but does not have its disadvantages. In conclusion, we present an interesting fact about the use of various types of weight importance in the problem of neural network pruning.