# Empirical investigations on WVA structural issues

arXiv: https://arxiv.org/abs/2208.05791 · HTML: https://mlsanity.com/paper/2208.05791/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 11 Aug 2022.

## Builds on (most cited first, 15 of 15)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Revisiting Natural Gradient for Deep Networks](https://mlsanity.com/paper/1301.3584/) (2013)
- [Differentiable plasticity: training plastic neural networks with backpropagation](https://mlsanity.com/paper/1804.02464/) (2018)
- [Hyperbolic Discounting and Learning over Multiple Horizons](https://mlsanity.com/paper/1902.06865/) (2019)
- [On Quadratic Penalties in Elastic Weight Consolidation](https://mlsanity.com/paper/1712.03847/) (2017)
- [Understanding Generalization through Visualizations](https://mlsanity.com/paper/1906.03291/) (2019)
- [Addressing catastrophic forgetting for medical domain expansion](https://mlsanity.com/paper/2103.13511/) (2021)
- [Towards continuous learning for glioma segmentation with elastic weight consolidation](https://mlsanity.com/paper/1909.11479/) (2019)
- [Enabling Continual Learning with Differentiable Hebbian Plasticity](https://mlsanity.com/paper/2006.16558/) (2020)
- [Sequential Domain Adaptation through Elastic Weight Consolidation for Sentiment Analysis](https://mlsanity.com/paper/2007.01189/) (2020)
- [Stabilizing Elastic Weight Consolidation method in practical ML tasks and using weight importances for neural network pruning](https://mlsanity.com/paper/2109.10021/) (2021)
- [Natural Way to Overcome the Catastrophic Forgetting in Neural Networks](https://mlsanity.com/paper/2005.07107/) (2020)

## Abstract (arXiv)

In this paper we want to present the results of empirical verification of some issues concerning the methods for overcoming catastrophic forgetting in neural networks. First, in the introduction, we will try to describe in detail the problem of catastrophic forgetting and methods for overcoming it for those who are not yet familiar with this topic. Then we will discuss the essence and limitations of the WVA method which we presented in previous papers. Further, we will touch upon the issues of applying the WVA method to gradients or optimization steps of weights, choosing the optimal attenuation function in this method, as well as choosing the optimal hyper-parameters of the method depending on the number of tasks in sequential training of neural networks.