mlsanityData to 7 Oct 2026

Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization

arXiv 2201.12250 ↗First posted 28 Jan 2022cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

Second-order optimizers are thought to hold the potential to speed up neural network training, but due to the enormous size of the curvature matrix, they typically require approximations to be computationally tractable. The most successful family of approximations are Kronecker-Factored, block-diagonal curvature estimates (KFAC). Here, we combine tools from prior work to evaluate exact second-order updates with careful ablations to establish a surprising result: Due to its approximations, KFAC is not closely related to second-order updates, and in particular, it significantly outperforms true second-order updates. This challenges widely held believes and immediately raises the question why KFAC performs so well. Towards answering this question we present evidence strongly suggesting that KFAC approximates a first-order algorithm, which performs gradient descent on neurons rather than weights. Finally, we show that this optimizer often improves over KFAC in terms of computational cost and data-efficiency.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 47 of its 78 references that are in the map. First posted on arXiv 28 Jan 2022.

Builds on

47 of 78 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  8. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  9. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  10. 2014New insights and perspectives on the natural gradient methodcited by 435
  11. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  12. 2017Variational Continual Learningcited by 335
35 more
  1. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  2. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  3. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 197
  4. 2018Recasting Gradient-Based Meta-Learning as Hierarchical Bayescited by 180
  5. 2013Revisiting Natural Gradient for Deep Networkscited by 157
  6. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 143
  7. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 139
  8. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 138
  9. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  10. 2017Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximationcited by 119
  11. 2018Three Mechanisms of Weight Decay Regularizationcited by 112
  12. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
  13. 2017Noisy Natural Gradient as Variational Inferencecited by 94
  14. 2014How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagationcited by 77
  15. 2019Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networkscited by 76
  16. 2020Practical Quasi-Newton Methods for Training Deep Neural Networkscited by 67
  17. 2016Training Neural Networks Without Gradients: A Scalable ADMM Approachcited by 61
  18. 2020A Theoretical Framework for Target Propagationcited by 56
  19. 2017Unbiased Online Recurrent Optimizationcited by 46
  20. 2015Natural Neural Networkscited by 44
  21. 2020Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processescited by 43
  22. 2018Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networkscited by 37
  23. 2019Efficient Subsampled Gauss-Newton and Natural Gradient Methods for Training Neural Networkscited by 32
  24. 2019Pathological spectra of the Fisher information metric and its variants in deep neural networkscited by 32
  25. 2013Riemannian metrics for neural networks I: feedforward networkscited by 29
  26. 2018Approximating Real-Time Recurrent Learning with Random Kronecker Factorscited by 29
  27. 2017Online Natural Gradient as a Kalman Filtercited by 25
  28. 2019Optimal Kronecker-Sum Approximation of Real Time Recurrent Learningcited by 21
  29. 2021ViViT: Curvature access through the generalized Gauss-Newton's low-rank structurecited by 16
  30. 2021LocoProp: Enhancing BackProp via Local Loss Optimizationcited by 15
  31. 2019Modular Block-diagonal Curvature Approximations for Feedforward Architecturescited by 13
  32. 2018Bayesian filtering unifies adaptive and non-adaptive neural network optimization methodscited by 9
  33. 2017Proximal Backpropagationcited by 7
  34. 2017True Asymptotic Natural Gradient Optimizationcited by 5
  35. 2016Practical Riemannian Neural Networkscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 8; 2024: 7; 2025: 10; 2026: 3.

29 papers.

Sort:

  1. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  2. 2024A Kaczmarz-inspired approach to accelerate the optimization of neural network wavefunctionscited by 24
  3. 2022Random initialisations performing above chance and how to find themcited by 23
  4. 2023FedLPA: One-shot Federated Learning with Layer-Wise Posterior Aggregationcited by 22
  5. 2024Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networkscited by 21
  6. 2025Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditionercited by 21
  7. 2024An Improved Empirical Fisher Approximation for Natural Gradient Descentcited by 15
  8. 2023On the Parameterization of Second-Order Optimization Effective Towards the Infinite Widthcited by 14
  9. 2024AdaFisher: Adaptive Second Order Optimization via Fisher Informationcited by 14
  10. 2023Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimizationcited by 13
  11. 2024Fast and Effective Weight Update for Pruned Large Language Modelscited by 11
  12. 2023Approximately Bayes-Optimal Pseudo Label Selectioncited by 10
17 more
  1. 2025Position: Curvature Matrices Should Be Democratized via Linear Operatorscited by 9
  2. 2025Online Curvature-Aware Replay: Leveraging mathbf2nd Order Information for Online Continual Learningcited by 8
  3. 2023Riemannian Laplace Approximation with the Fisher Metriccited by 7
  4. 2025Dual Natural Gradient Descent for Scalable Training of Physics-Informed Neural Networkscited by 6
  5. 2026Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectorycited by 5
  6. 2024Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagationcited by 4
  7. 2025Converge Faster, Talk Less: Hessian-Informed Federated Zeroth-Order Optimizationcited by 3
  8. 2024Debiasing Mini-Batch Quadratics for Applications in Deep Learningcited by 2
  9. 2025Better Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functionscited by 2
  10. 2025Adam or Gauss-Newton? A Comparative Study In Terms of Basis Alignment and SGD Noisecited by 2
  11. 2026Rank-1 Approximation of Inverse Fisher for Natural Policy Gradients in Deep Reinforcement Learningcited by 2
  12. 2023Eva: A General Vectorized Approximation Framework for Second-order Optimizationcited by 1
  13. 2025MAC: An Efficient Gradient Preconditioning using Mean Activation Approximated Curvaturecited by 1
  14. 2023Analysis and Comparison of Two-Level KFAC Methods for Training Deep Neural Networkscited by 0
  15. 2025An Adaptive Method Stabilizing Activations for Enhanced Generalizationcited by 0
  16. 2025NysAct: A Scalable Preconditioned Gradient Descent using Nystrom Approximationcited by 0
  17. 2026Randomized Advantage Transformation (RAT): Computing Natural Policy Gradients via Direct Backpropagationcited by 0