mlsanityData to 7 Oct 2026

Limitations of the Empirical Fisher Approximation for Natural Gradient Descent

arXiv 1905.12558 ↗First posted 29 May 2019cs.LG stat.MLcited by 213

Not on the map. Why?

Abstract (arXiv)

Natural gradient descent, which preconditions a gradient descent update with the Fisher information matrix of the underlying statistical model, is a way to capture partial second-order information. Several highly visible works have advocated an approximation known as the empirical Fisher, drawing connections between approximate second-order methods and heuristics like Adam. We dispute this argument by showing that the empirical Fisher---unlike the Fisher---does not generally capture second-order information. We further argue that the conditions under which the empirical Fisher approaches the Fisher (and the Hessian) are unlikely to be met in practice, and that, even on simple optimization problems, the pathologies of the empirical Fisher can have undesirable effects.

Key facts. Cited by 213 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 31; 2022: 17; 2023: 35; 2024: 32; 2025: 37; 2026: 61.

213 papers.

Sort:

By area:

  1. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  2. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  3. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  4. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  5. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  6. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 112
  7. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
  8. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 78
  9. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 69
  10. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
  11. 2021M-FAC: Efficient Matrix-Free Approximations of Second-Order Informationcited by 54
  12. 2024A New Perspective on Shampoo's Preconditionercited by 54
38 more
  1. 2022A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awarenesscited by 49
  2. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  3. 2021Interactive Label Cleaning with Example-based Explanationscited by 48
  4. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 47
  5. 2021Natural continual learning: success is a journey, not (just) a destinationcited by 46
  6. 2022Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximationscited by 42
  7. 2025In Search of Adam's Secret Saucecited by 40
  8. 2021Strength of Minibatch Noise in SGDcited by 39
  9. 2023Merging by Matching Models in Task Parameter Subspacescited by 36
  10. 2023The LLM Surgeoncited by 36
  11. 2024Bayesian Reward Models for LLM Alignmentcited by 35
  12. 2021Sketching Curvature for Efficient Out-of-Distribution Detection for Deep Neural Networkscited by 32
  13. 2024Influence Functions for Scalable Data Attribution in Diffusion Modelscited by 31
  14. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 29
  15. 2022Acceleration of Federated Learning with Alleviated Forgetting in Local Trainingcited by 29
  16. 2023Elastic Weight Removal for Faithful and Abstractive Dialogue Generationcited by 29
  17. 2022Unifying Approaches in Active Learning and Active Sampling via Fisher Information and Information-Theoretic Quantitiescited by 28
  18. 2023Directions of Curvature as an Explanation for Loss of Plasticitycited by 26
  19. 2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25
  20. 2021Tensor Normal Training for Deep Learning Modelscited by 24
  21. 2021On the Variance of the Fisher Information for Deep Learningcited by 22
  22. 2023FedLPA: One-shot Federated Learning with Layer-Wise Posterior Aggregationcited by 22
  23. 2024Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networkscited by 21
  24. 2025Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditionercited by 21
  25. 2023ASDL: A Unified Interface for Gradient Preconditioning in PyTorchcited by 20
  26. 2025Model-Preserving Adaptive Roundingcited by 20
  27. 2022Sampling-based inference for large linear models, with application to linearised Laplacecited by 18
  28. 2023Jumping through Local Minima: Quantization in the Loss Landscape of Vision Transformerscited by 18
  29. 2024A Second-Order Perspective on Model Compositionality and Incremental Learningcited by 18
  30. 2022Hybrid quantum ResNet for car classification and its hyperparameter optimizationcited by 17
  31. 2023Federated Learning as Variational Inference: A Scalable Expectation Propagation Approachcited by 17
  32. 2024FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher informationcited by 17
  33. 2021ViViT: Curvature access through the generalized Gauss-Newton's low-rank structurecited by 16
  34. 2023The Geometry of Neural Nets' Parameter Spaces Under Reparametrizationcited by 16
  35. 2025Efficient Model Editing with Task-Localized Sparse Fine-tuningcited by 16
  36. 2021Lifelong Learning with Sketched Structural Regularizationcited by 15
  37. 2021LocoProp: Enhancing BackProp via Local Loss Optimizationcited by 15
  38. 2021Provable Continual Learning via Sketched Jacobian Approximationscited by 15

50 most cited of 213 All 213 (JSON)