mlsanityData to 7 Oct 2026

Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC

arXiv 2312.05705 ↗First posted 9 Dec 2023cs.LG stat.MLcited by 9

Not on the map. Why?

Abstract (arXiv)

Second-order methods such as KFAC can be useful for neural net training. However, they are often memory-inefficient since their preconditioning Kronecker factors are dense, and numerically unstable in low precision as they require matrix inversion or decomposition. These limitations render such methods unpopular for modern mixed-precision training. We address them by (i) formulating an inverse-free KFAC update and (ii) imposing structures in the Kronecker factors, resulting in structured inverse-free natural gradient descent (SINGD). On modern neural networks, we show that SINGD is memory-efficient and numerically robust, in contrast to KFAC, and often outperforms AdamW even in half precision. Our work closes a gap between first- and second-order methods in modern low-precision training.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 42 of its 59 references that are in the map. First posted on arXiv 9 Dec 2023.

Builds on

42 of 59 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  8. 2017Mixed Precision Trainingcited by 1,135
  9. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  10. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  11. 2014New insights and perspectives on the natural gradient methodcited by 435
  12. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
30 more
  1. 2023Studying Large Language Model Generalization with Influence Functionscited by 311
  2. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 261
  3. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  4. 2022Patches Are All You Need?cited by 201
  5. 2020The Computational Limits of Deep Learningcited by 188
  6. 2019Practical Deep Learning with Bayesian Principlescited by 158
  7. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 143
  8. 2022If Influence Functions are the Answer, Then What is the Question?cited by 142
  9. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 139
  10. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 138
  11. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  12. 2023RepViT: Revisiting Mobile CNN From ViT Perspectivecited by 123
  13. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 116
  14. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
  15. 2019EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasiscited by 98
  16. 2022Global Context Vision Transformerscited by 98
  17. 2021The Bayesian Learning Rulecited by 95
  18. 2017Noisy Natural Gradient as Variational Inferencecited by 94
  19. 2017Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Modelscited by 65
  20. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  21. 2018Fast yet Simple Natural-Gradient Descent for Variational Inference in Complex Modelscited by 37
  22. 2020Handling the Positive-Definite Constraint in the Bayesian Learning Rulecited by 31
  23. 2022Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasetscited by 28
  24. 2020Optimization of Graph Neural Networks with Natural Gradient Descentcited by 26
  25. 2022PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matricescited by 24
  26. 2021Tractable structured natural gradient descent using local parameterizationscited by 22
  27. 2018Preconditioner on Matrix Lie Group for SGDcited by 15
  28. 2023Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learningcited by 10
  29. 2022Black Box Lie Group Preconditioners for SGDcited by 8
  30. 2023Convolutions and More as Einsum: A Tensor Network Perspective with Advances for Second-Order Methodscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 5.

9 papers.

  1. 2024Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networkscited by 21
  2. 2025Position: Curvature Matrices Should Be Democratized via Linear Operatorscited by 9
  3. 2025Beyond the Mean: Fisher-Orthogonal Projection for Natural Gradient Descent in Large Batch Trainingcited by 5
  4. 2025Optimization Guarantees for Square-Root Natural-Gradient Variational Inferencecited by 3
  5. 2026Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvaturecited by 3
  6. 2026Efficient Bilevel Optimization with KFAC-Based Hypergradientscited by 1
  7. 2026Layerwise LQR for Geometry-Aware Optimization of Deep Networkscited by 1
  8. 2026Second-Order Muon Done Right: A Principled Marriage of Spectral Geometry and Curvaturecited by 1
  9. 2026Detecting Contaminated Code-Generation Prompt Batches via Influence Functionscited by 0