# On the Computation of the Fisher Information in Continual Learning

arXiv: https://arxiv.org/abs/2502.11756 · HTML: https://mlsanity.com/paper/2502.11756/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 20 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Builds on (most cited first, 13 of 13)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [Limitations of the Empirical Fisher Approximation for Natural Gradient Descent](https://mlsanity.com/paper/1905.12558/) (2019)
- [Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1805.07810/) (2018)
- [Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting](https://mlsanity.com/paper/1802.02950/) (2018)
- [PyCIL: A Python Toolbox for Class-Incremental Learning](https://mlsanity.com/paper/2112.12533/) (2021)
- [On Quadratic Penalties in Elastic Weight Consolidation](https://mlsanity.com/paper/1712.03847/) (2017)
- [Natural continual learning: success is a journey, not (just) a destination](https://mlsanity.com/paper/2106.08085/) (2021)
- [Avalanche: A PyTorch Library for Deep Continual Learning](https://mlsanity.com/paper/2302.01766/) (2023)
- [Elastic Weight Consolidation (EWC): Nuts and Bolts](https://mlsanity.com/paper/2105.04093/) (2021)
- [Two Complementary Perspectives to Continual Learning: Ask Not Only What to Optimize, But Also How](https://mlsanity.com/paper/2311.04898/) (2023)

## Built on by (most cited first, 13 of 13)

- [Full-Parameter Continual Pretraining of Gemma2: Insights into Fluency and Domain Knowledge](https://mlsanity.com/paper/2505.05946/) (2025)
- [HERMAN: Hierarchical Representation Matching for CLIP-based Class-Incremental Learning](https://mlsanity.com/paper/2509.22645/) (2025)
- [Revisiting Weight Regularization for Low-Rank Continual Learning](https://mlsanity.com/paper/2602.17559/) (2026)
- [SLE-FNO: Single-Layer Extensions for Task-Agnostic Continual Learning in Fourier Neural Operators](https://mlsanity.com/paper/2603.20410/) (2026)
- [Sparsity-Driven Plasticity in Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2508.06871/) (2025)
- [Fisher-Orthogonal Projected Natural Gradient Descent for Continual Learning](https://mlsanity.com/paper/2601.12816/) (2026)
- [Self-Consolidating Language Models: Continual Knowledge Incorporation from Context](https://mlsanity.com/paper/2605.07076/) (2026)
- [Affect and Effect: Limitations of regularisation-based continual learning in EEG-based emotion classification](https://mlsanity.com/paper/2601.07858/) (2026)
- [BRAIN: Bias-Mitigation Continual Learning Approach to Vision-Brain Understanding](https://mlsanity.com/paper/2508.18187/) (2025)
- [Unbiased Online Curvature Approximation for Regularized Graph Continual Learning](https://mlsanity.com/paper/2509.12727/) (2025)
- [Balancing Stability and Plasticity in Sequentially Trained Early-Exiting Neural Networks](https://mlsanity.com/paper/2605.05358/) (2026)
- [Normative Loss Landscape Navigation: A Trajectory-Based Approach to Mitigating Forgetting in Incremental Learning](https://mlsanity.com/paper/2609.35926/) (2026)
- [Do Artificial Neural Networks Need Dendritic Nanotubes? Two Reasons for Shrinking Importance Estimates in Continual Learning](https://mlsanity.com/paper/2610.14258/) (2026)

## Abstract (arXiv)

One of the most popular methods for continual learning with deep neural networks is Elastic Weight Consolidation (EWC), which involves computing the Fisher Information. The exact way in which the Fisher Information is computed is however rarely described, and multiple different implementations for it can be found online. This blog post discusses and empirically compares several often-used implementations, which highlights that many currently reported results for EWC could likely be improved by changing the way the Fisher Information is computed.