# On the Variance of the Fisher Information for Deep Learning

arXiv: https://arxiv.org/abs/2107.04205 · HTML: https://mlsanity.com/paper/2107.04205/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 44 references that are in the map.
- First posted on arXiv 9 Jul 2021.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [Limitations of the Empirical Fisher Approximation for Natural Gradient Descent](https://mlsanity.com/paper/1905.12558/) (2019)
- [Revisiting Natural Gradient for Deep Networks](https://mlsanity.com/paper/1301.3584/) (2013)
- [Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach](https://mlsanity.com/paper/1806.01316/) (2018)
- [An elementary introduction to information geometry](https://mlsanity.com/paper/1808.08271/) (2018)
- [Riemannian metrics for neural networks I: feedforward networks](https://mlsanity.com/paper/1303.0818/) (2013)
- [Fisher Information and Natural Gradient Learning of Random Deep Networks](https://mlsanity.com/paper/1808.07172/) (2018)
- [Estimating Fisher Information Matrix in Latent Variable Models based on the Score Function](https://mlsanity.com/paper/1909.06094/) (2019)

## Built on by (most cited first, 22 of 22)

- [Weighted Ensemble Models Are Strong Continual Learners](https://mlsanity.com/paper/2312.08977/) (2023)
- [Exploring Temporal Information Dynamics in Spiking Neural Networks](https://mlsanity.com/paper/2211.14406/) (2022)
- [A numerical approximation method for the Fisher-Rao distance between multivariate normal distributions](https://mlsanity.com/paper/2302.08175/) (2023)
- [Trade-Offs of Diagonal Fisher Information Matrix Estimators](https://mlsanity.com/paper/2402.05379/) (2024)
- [TEAM: Topological Evolution-aware Framework for Traffic Forecasting--Extended Version](https://mlsanity.com/paper/2410.19192/) (2024)
- [OASIS: Online Sample Selection for Continual Visual Instruction Tuning](https://mlsanity.com/paper/2506.02011/) (2025)
- [Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Sampling](https://mlsanity.com/paper/2410.15143/) (2024)
- [Structural Neural Additive Models: Enhanced Interpretable Machine Learning](https://mlsanity.com/paper/2302.09275/) (2023)
- [A Differential Geometric View and Explainability of GNN on Evolving Graphs](https://mlsanity.com/paper/2403.06425/) (2024)
- [Fisher Mask Nodes for Language Model Merging](https://mlsanity.com/paper/2403.09891/) (2024)
- [Adapting In-Domain Few-Shot Segmentation to New Domains without Source Domain Retraining](https://mlsanity.com/paper/2504.21414/) (2025)
- [Uncovering Model Processing Strategies with Non-Negative Per-Example Fisher Factorization](https://mlsanity.com/paper/2310.04649/) (2023)
- [Generalizable and Stable Finetuning of Pretrained Language Models on Low-Resource Texts](https://mlsanity.com/paper/2403.12918/) (2024)
- [Confidence Sets under Generalized Self-Concordance](https://mlsanity.com/paper/2301.00260/) (2022)
- [AdaBet: Gradient-free Layer Selection for Efficient Training of Deep Neural Networks](https://mlsanity.com/paper/2510.03101/) (2025)
- [Robust Ultra Low-Bit Post-Training Quantization via Stable Diagonal Curvature Estimate](https://mlsanity.com/paper/2604.13806/) (2026)
- [Mitigating Catastrophic Forgetting in Long Short-Term Memory Networks](https://mlsanity.com/paper/2305.17244/) (2023)
- [Alpha and Prejudice: Improving α-sized Worst-case Fairness via Intrinsic Reweighting](https://mlsanity.com/paper/2411.03068/) (2024)
- [Deterministic Bounds and Random Estimates of Metric Tensors on Neuromanifolds](https://mlsanity.com/paper/2505.13614/) (2025)
- [Zero-Direction Probing: A Linear-Algebraic Framework for Deep Analysis of Large-Language-Model Drift](https://mlsanity.com/paper/2508.06776/) (2025)
- [Confidence-Calibrating Regularization for Robust Brain MRI Segmentation Under Domain Shift](https://mlsanity.com/paper/2509.23176/) (2025)
- [Masking Radar Cognition under Adversarial Surveillance: A Distributional Privacy Framework](https://mlsanity.com/paper/2609.07428/) (2026)

## Abstract (arXiv)

In the realm of deep learning, the Fisher information matrix (FIM) gives novel insights and useful tools to characterize the loss landscape, perform second-order optimization, and build geometric learning theories. The exact FIM is either unavailable in closed form or too expensive to compute. In practice, it is almost always estimated based on empirical samples. We investigate two such estimators based on two equivalent representations of the FIM -- both unbiased and consistent. Their estimation quality is naturally gauged by their variance given in closed form. We analyze how the parametric structure of a deep neural network can affect the variance. The meaning of this variance measure and its upper bounds are then discussed in the context of deep learning.