mlsanityData to 7 Oct 2026

Trade-Offs of Diagonal Fisher Information Matrix Estimators

arXiv 2402.05379 ↗First posted 8 Feb 2024cs.LG stat.MLcited by 10

Not on the map. Why?

Abstract (arXiv)

The Fisher information matrix can be used to characterize the local geometry of the parameter space of neural networks. It elucidates insightful theories and useful tools to understand and optimize neural networks. Given its high computational cost, practitioners often use random estimators and evaluate only the diagonal entries. We examine two popular estimators whose accuracy and sample complexity depend on their associated variances. We derive bounds of the variances and instantiate them in neural networks for regression and classification. We navigate trade-offs for both estimators based on analytical and numerical studies. We find that the variance quantities depend on the non-linearity wrt different parameter groups and should not be neglected when estimating the Fisher information.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 21 of its 49 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

21 of 49 references.

References · 21

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  3. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  4. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  5. 2014New insights and perspectives on the natural gradient methodcited by 435
  6. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  7. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  8. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  9. 2013Revisiting Natural Gradient for Deep Networkscited by 157
  10. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  11. 2018Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approachcited by 89
  12. 2019BackPACK: Packing more into backpropcited by 77
9 more
  1. 2019Pathological spectra of the Fisher information metric and its variants in deep neural networkscited by 32
  2. 2013Riemannian metrics for neural networks I: feedforward networkscited by 29
  3. 2018Fisher Information and Natural Gradient Learning of Random Deep Networkscited by 28
  4. 2021Tensor Normal Training for Deep Learning Modelscited by 24
  5. 2020Understanding Approximate Fisher Information for Fast Convergence of Natural Gradient Descent in Wide Neural Networkscited by 22
  6. 2021On the Variance of the Fisher Information for Deep Learningcited by 22
  7. 2023The Memory Perturbation Equation: Understanding Model's Sensitivity to Datacited by 16
  8. 2019A Geometric Modeling of Occam's Razor in Deep Learningcited by 8
  9. 2022HesScale: Scalable Computation of Hessian Diagonalscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025Position: Curvature Matrices Should Be Democratized via Linear Operatorscited by 9
  2. 2025Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Modelscited by 7
  3. 2026SLE-FNO: Single-Layer Extensions for Task-Agnostic Continual Learning in Fourier Neural Operatorscited by 2
  4. 2025What Scalable Second-Order Information Knows for Pruning at Initializationcited by 1
  5. 2025Deterministic Bounds and Random Estimates of Metric Tensors on Neuromanifoldscited by 0
  6. 2026Robust Ultra Low-Bit Post-Training Quantization via Stable Diagonal Curvature Estimatecited by 0
  7. 2026Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Expertscited by 0
  8. 2026MAdam: Metric-Aware Multi-Objective Adamcited by 0
  9. 2026Fisher-Guided Progressive Parameter Selection for Adaptive Fine-Tuningcited by 0
  10. 2026Normative Loss Landscape Navigation: A Trajectory-Based Approach to Mitigating Forgetting in Incremental Learningcited by 0