Data to 7 Oct 2026

Optimizing Neural Networks with Kronecker-factored Approximate Curvature

arXiv 1503.05671 ↗First posted 19 Mar 2015cs.LG cs.NE stat.MLcited by 776

Not on the map. Why?

Abstract (arXiv)

We propose an efficient method for approximating natural gradient descent in neural networks which we call Kronecker-Factored Approximate Curvature (K-FAC). K-FAC is based on an efficiently invertible approximation of a neural network's Fisher information matrix which is neither diagonal nor low-rank, and in some cases is completely non-sparse. It is derived by approximating various large blocks of the Fisher (corresponding to entire layers) as being the Kronecker product of two much smaller matrices. While only several times more expensive to compute than the plain stochastic gradient, the updates produced by K-FAC make much more progress optimizing the objective, which results in an algorithm that can be much faster than stochastic gradient descent with momentum in practice. And unlike some previously proposed approximate natural-gradient/Newton methods which use high-quality non-diagonal curvature matrices (such as Hessian-free optimization), K-FAC works very well in highly stochastic optimization regimes. This is because the cost of storing and inverting K-FAC's approximation to the curvature matrix does not depend on the amount of data used to estimate it, which is a feature typically associated only with diagonal or low-rank approximations to the curvature matrix.

Key facts. Cited by 776 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Mar 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 82; 2022: 80; 2023: 95; 2024: 119; 2025: 156; 2026: 243.

776 papers.

Sort:

By area:

2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
2021A Survey of Uncertainty in Deep Neural Networkscited by 675
2021Merging Models with Fisher-Weighted Averagingcited by 613
2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
2023Symbolic Discovery of Optimization Algorithmscited by 423
2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 179
2022If Influence Functions are the Answer, Then What is the Question?cited by 142
2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134

38 more

2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
2018Multi-Agent Generative Adversarial Imitation Learningcited by 99
2024What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functionscited by 88
2022Deep Unlearning via Randomized Conditionally Independent Hessianscited by 83
2025NorMuon: Making Muon more efficient and scalablecited by 74
2024Continual Learning and Catastrophic Forgettingcited by 73
2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 69
2024Fisher Flow Matching for Generative Modeling over Discrete Datacited by 68
2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
2021Stochastic Training is Not Necessary for Generalizationcited by 63
2021Estimating High Order Gradients of the Data Distribution by Denoisingcited by 63
2021Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Timecited by 62
2022Fast Finite Width Neural Tangent Kernelcited by 62
2022Adaptive Second Order Coresets for Data-efficient Machine Learningcited by 60
2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 59
2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
2022Bayesian Model Selection, the Marginal Likelihood, and Generalizationcited by 56
2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
2025Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspectivecited by 55
2025ASGO: Adaptive Structured Gradient Optimizationcited by 55
2021M-FAC: Efficient Matrix-Free Approximations of Second-Order Informationcited by 54
2024A New Perspective on Shampoo's Preconditionercited by 54
2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 51
2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
2021Interactive Label Cleaning with Example-based Explanationscited by 48
2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 47
2021Natural continual learning: success is a journey, not (just) a destinationcited by 46
2022A Self-Attention Ansatz for Ab-initio Quantum Chemistrycited by 45
2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 44
2023Towards Sustainable Learning: Coresets for Data-efficient Deep Learningcited by 44
2023Incorporating Neuro-Inspired Adaptability for Continual Learning in Artificial Intelligencecited by 43

50 most cited of 776 All 776 (JSON)