mlsanityData to 7 Oct 2026

Correlation of the importances of neural network weights calculated by modern methods of overcoming catastrophic forgetting

arXiv 2211.17012 ↗First posted 24 Oct 2022cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Following the invention in 2017 of the EWC method, several methods have been proposed to calculate the importance of neural network weights for use in the EWC method. Despite the significant difference in calculating the importance of weights, they all proved to be effective. Accordingly, a reasonable question arises as to how similar the importances of the weights calculated by different methods. To answer this question, we calculated layer-by-layer correlations of the importance of weights calculated by all those methods. As a result, it turned out that the importances of several of the methods correlated with each other quite strongly and we were able to present an explanation for such a correlation. At the same time, for other methods, the correlation can vary from strong on some layers of the network to negative on other layers. Which raises a reasonable question: why, despite the very different calculation methods, all those importances allow EWC method to overcome the catastrophic forgetting of neural networks perfectly?

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 15 of its 25 references that are in the map. First posted on arXiv 24 Oct 2022.

Builds on

15 of 25 references.

References · 15

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  3. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  4. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  5. 2013Revisiting Natural Gradient for Deep Networkscited by 157
  6. 2018Differentiable plasticity: training plastic neural networks with backpropagationcited by 83
  7. 2019Hyperbolic Discounting and Learning over Multiple Horizonscited by 58
  8. 2017On Quadratic Penalties in Elastic Weight Consolidationcited by 54
  9. 2019Understanding Generalization through Visualizationscited by 54
  10. 2021Addressing catastrophic forgetting for medical domain expansioncited by 12
  11. 2019Towards continuous learning for glioma segmentation with elastic weight consolidationcited by 11
  12. 2020Enabling Continual Learning with Differentiable Hebbian Plasticitycited by 8
3 more
  1. 2020Sequential Domain Adaptation through Elastic Weight Consolidation for Sentiment Analysiscited by 8
  2. 2021Stabilizing Elastic Weight Consolidation method in practical ML tasks and using weight importances for neural network pruningcited by 8
  3. 2020Natural Way to Overcome the Catastrophic Forgetting in Neural Networkscited by 5

Built on by

Not cited yet.