mlsanityData to 14 Oct 2026

Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis

arXiv 1806.03884 ↗First posted 11 Jun 2018cs.LG stat.MLcited by 146

Not on the map. Why?

Abstract (arXiv)

Optimization algorithms that leverage gradient covariance information, such as variants of natural gradient descent (Amari, 1998), offer the prospect of yielding more effective descent directions. For models with many parameters, the covariance matrix they are based on becomes gigantic, making them inapplicable in their original form. This has motivated research into both simple diagonal approximations and more sophisticated factored approximations such as KFAC (Heskes, 2000; Martens & Grosse, 2015; Grosse & Martens, 2016). In the present work we draw inspiration from both to propose a novel approximation that is provably better than KFAC and amendable to cheap partial updates. It consists in tracking a diagonal variance, not in parameter coordinates, but in a Kronecker-factored eigenbasis, in which the diagonal approximation is likely to be more effective. Experiments show improvements over KFAC in optimization speed for several deep network architectures.

Key facts. Cited by 146 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 10; 2023: 16; 2024: 35; 2025: 30; 2026: 44.

146 papers.

Sort:

By area:

  1. 2021A Survey of Uncertainty in Deep Neural Networkscited by 680
  2. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
  3. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  4. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
  5. 2023DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Modelscited by 123
  6. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  7. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 77
  8. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  9. 2024A New Perspective on Shampoo's Preconditionercited by 59
  10. 2024Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizercited by 53
  11. 2023Merging by Matching Models in Task Parameter Subspacescited by 36
  12. 2024Training Data Attribution via Approximate Unrolled Differentiationcited by 34
38 more
  1. 2024Influence Functions for Scalable Data Attribution in Diffusion Modelscited by 33
  2. 2025Structured Preconditioners in Adaptive Optimization: A Unified Analysiscited by 33
  3. 2024Most Influential Subset Selection: Challenges, Promises, and Beyondcited by 32
  4. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 30
  5. 2025COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMscited by 30
  6. 2025The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Trainingcited by 30
  7. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  8. 2024A Comprehensive Survey on Evidential Deep Learning and Its Applicationscited by 25
  9. 2024Editable Concept Bottleneck Modelscited by 23
  10. 2025Model-Preserving Adaptive Roundingcited by 23
  11. 2025Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditionercited by 23
  12. 2023FedLPA: One-shot Federated Learning with Layer-Wise Posterior Aggregationcited by 22
  13. 2024Data Attribution for Text-to-Image Models by Unlearning Synthesized Imagescited by 22
  14. 2025MAGIC: Near-Optimal Data Attribution for Deep Learningcited by 22
  15. 2023ASDL: A Unified Interface for Gradient Preconditioning in PyTorchcited by 21
  16. 2021KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networkscited by 20
  17. 2024SWAN: SGD with Normalization and Whitening Enables Stateless LLM Trainingcited by 20
  18. 2026The Newton-Muon Optimizercited by 18
  19. 2024Reparameterization invariance in approximate Bayesian inferencecited by 17
  20. 2025Understanding and Improving Shampoo and SOAP via Kullback-Leibler Minimizationcited by 16
  21. 2024An Improved Empirical Fisher Approximation for Natural Gradient Descentcited by 15
  22. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  23. 2025Bayesian Influence Functions for Hessian-Free Data Attributioncited by 14
  24. 2025Towards Quantifying the Hessian Structure of Neural Networkscited by 13
  25. 2021Second-Order Neural ODE Optimizercited by 11
  26. 2022DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learningcited by 11
  27. 2026FISMO: Fisher-Structured Momentum-Orthogonalized Optimizercited by 11
  28. 2025Better Training Data Attribution via Better Inverse Hessian-Vector Productscited by 10
  29. 2026Mechanistic Data Attribution: Tracing the Training Origins of Interpretable LLM Unitscited by 10
  30. 2022Efficient Approximations of the Fisher Matrix in Neural Networks using Kronecker Product Singular Value Decompositioncited by 9
  31. 2022Amortized Proximal Optimizationcited by 9
  32. 2023Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFACcited by 9
  33. 2024Bayesian Parameter-Efficient Fine-Tuning for Overcoming Catastrophic Forgettingcited by 9
  34. 2024On Evaluating Explanation Utility for Human-AI Decision Making in NLPcited by 9
  35. 2024Correcting Large Language Model Behavior via Influence Functioncited by 9
  36. 2025Position: Curvature Matrices Should Be Democratized via Linear Operatorscited by 9
  37. 2022A Mini-Block Fisher Method for Deep Neural Networkscited by 8
  38. 2023Natural Gradient Methods: Perspectives, Efficient-Scalable Approximations, and Analysiscited by 8

50 most cited of 146 All 146 (JSON)