# Correlation of the importances of neural network weights calculated by modern methods of overcoming catastrophic forgetting

arXiv: https://arxiv.org/abs/2211.17012 · HTML: https://mlsanity.com/paper/2211.17012/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 24 Oct 2022.

## Builds on (most cited first, 15 of 15)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Revisiting Natural Gradient for Deep Networks](https://mlsanity.com/paper/1301.3584/) (2013)
- [Differentiable plasticity: training plastic neural networks with backpropagation](https://mlsanity.com/paper/1804.02464/) (2018)
- [Hyperbolic Discounting and Learning over Multiple Horizons](https://mlsanity.com/paper/1902.06865/) (2019)
- [On Quadratic Penalties in Elastic Weight Consolidation](https://mlsanity.com/paper/1712.03847/) (2017)
- [Understanding Generalization through Visualizations](https://mlsanity.com/paper/1906.03291/) (2019)
- [Addressing catastrophic forgetting for medical domain expansion](https://mlsanity.com/paper/2103.13511/) (2021)
- [Towards continuous learning for glioma segmentation with elastic weight consolidation](https://mlsanity.com/paper/1909.11479/) (2019)
- [Enabling Continual Learning with Differentiable Hebbian Plasticity](https://mlsanity.com/paper/2006.16558/) (2020)
- [Sequential Domain Adaptation through Elastic Weight Consolidation for Sentiment Analysis](https://mlsanity.com/paper/2007.01189/) (2020)
- [Stabilizing Elastic Weight Consolidation method in practical ML tasks and using weight importances for neural network pruning](https://mlsanity.com/paper/2109.10021/) (2021)
- [Natural Way to Overcome the Catastrophic Forgetting in Neural Networks](https://mlsanity.com/paper/2005.07107/) (2020)

## Abstract (arXiv)

Following the invention in 2017 of the EWC method, several methods have been proposed to calculate the importance of neural network weights for use in the EWC method. Despite the significant difference in calculating the importance of weights, they all proved to be effective. Accordingly, a reasonable question arises as to how similar the importances of the weights calculated by different methods. To answer this question, we calculated layer-by-layer correlations of the importance of weights calculated by all those methods. As a result, it turned out that the importances of several of the methods correlated with each other quite strongly and we were able to present an explanation for such a correlation. At the same time, for other methods, the correlation can vary from strong on some layers of the network to negative on other layers. Which raises a reasonable question: why, despite the very different calculation methods, all those importances allow EWC method to overcome the catastrophic forgetting of neural networks perfectly?