mlsanityData to 7 Oct 2026

On the Variance of the Fisher Information for Deep Learning

arXiv 2107.04205 ↗First posted 9 Jul 2021cs.LG stat.MLcited by 22

Not on the map. Why?

Abstract (arXiv)

In the realm of deep learning, the Fisher information matrix (FIM) gives novel insights and useful tools to characterize the loss landscape, perform second-order optimization, and build geometric learning theories. The exact FIM is either unavailable in closed form or too expensive to compute. In practice, it is almost always estimated based on empirical samples. We investigate two such estimators based on two equivalent representations of the FIM -- both unbiased and consistent. Their estimation quality is naturally gauged by their variance given in closed form. We analyze how the parametric structure of a deep neural network can affect the variance. The meaning of this variance measure and its upper bounds are then discussed in the context of deep learning.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 14 of its 44 references that are in the map. First posted on arXiv 9 Jul 2021.

Builds on

14 of 44 references.

References · 14

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2016Layer Normalizationcited by 5,579
  5. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  6. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  7. 2014New insights and perspectives on the natural gradient methodcited by 435
  8. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  9. 2013Revisiting Natural Gradient for Deep Networkscited by 157
  10. 2018Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approachcited by 89
  11. 2018An elementary introduction to information geometrycited by 76
  12. 2013Riemannian metrics for neural networks I: feedforward networkscited by 29
2 more
  1. 2018Fisher Information and Natural Gradient Learning of Random Deep Networkscited by 28
  2. 2019Estimating Fisher Information Matrix in Latent Variable Models based on the Score Functioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 5; 2024: 7; 2025: 6; 2026: 2.

22 papers.

Sort:

  1. 2023Weighted Ensemble Models Are Strong Continual Learnerscited by 41
  2. 2022Exploring Temporal Information Dynamics in Spiking Neural Networkscited by 23
  3. 2023A numerical approximation method for the Fisher-Rao distance between multivariate normal distributionscited by 11
  4. 2024Trade-Offs of Diagonal Fisher Information Matrix Estimatorscited by 10
  5. 2024TEAM: Topological Evolution-aware Framework for Traffic Forecasting--Extended Versioncited by 10
  6. 2025OASIS: Online Sample Selection for Continual Visual Instruction Tuningcited by 8
  7. 2024Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Samplingcited by 7
  8. 2023Structural Neural Additive Models: Enhanced Interpretable Machine Learningcited by 6
  9. 2024A Differential Geometric View and Explainability of GNN on Evolving Graphscited by 4
  10. 2024Fisher Mask Nodes for Language Model Mergingcited by 4
  11. 2025Adapting In-Domain Few-Shot Segmentation to New Domains without Source Domain Retrainingcited by 4
  12. 2023Uncovering Model Processing Strategies with Non-Negative Per-Example Fisher Factorizationcited by 2
10 more
  1. 2024Generalizable and Stable Finetuning of Pretrained Language Models on Low-Resource Textscited by 2
  2. 2022Confidence Sets under Generalized Self-Concordancecited by 1
  3. 2025AdaBet: Gradient-free Layer Selection for Efficient Training of Deep Neural Networkscited by 1
  4. 2023Mitigating Catastrophic Forgetting in Long Short-Term Memory Networkscited by 0
  5. 2024Alpha and Prejudice: Improving α-sized Worst-case Fairness via Intrinsic Reweightingcited by 0
  6. 2025Deterministic Bounds and Random Estimates of Metric Tensors on Neuromanifoldscited by 0
  7. 2025Zero-Direction Probing: A Linear-Algebraic Framework for Deep Analysis of Large-Language-Model Driftcited by 0
  8. 2025Confidence-Calibrating Regularization for Robust Brain MRI Segmentation Under Domain Shiftcited by 0
  9. 2026Robust Ultra Low-Bit Post-Training Quantization via Stable Diagonal Curvature Estimatecited by 0
  10. 2026Masking Radar Cognition under Adversarial Surveillance: A Distributional Privacy Frameworkcited by 0