# Efficient Subsampled Gauss-Newton and Natural Gradient Methods for Training Neural Networks

arXiv: https://arxiv.org/abs/1906.02353 · HTML: https://mlsanity.com/paper/1906.02353/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jun 2019.

## Built on by (most cited first, 30 of 33)

- [Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization](https://mlsanity.com/paper/2201.12250/) (2022)
- [A Kaczmarz-inspired approach to accelerate the optimization of neural network wavefunctions](https://mlsanity.com/paper/2401.10190/) (2024)
- [Regularized Newton Method with Global O(1/k2) Convergence](https://mlsanity.com/paper/2112.02089/) (2021)
- [ASDL: A Unified Interface for Gradient Preconditioning in PyTorch](https://mlsanity.com/paper/2305.04684/) (2023)
- [An Improved Empirical Fisher Approximation for Natural Gradient Descent](https://mlsanity.com/paper/2406.06420/) (2024)
- [Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomization](https://mlsanity.com/paper/2505.12149/) (2025)
- [Achieving High Accuracy with PINNs via Energy Natural Gradients](https://mlsanity.com/paper/2302.13163/) (2023)
- [Exact Gauss-Newton Optimization for Training Deep Neural Networks](https://mlsanity.com/paper/2405.14402/) (2024)
- [Position: Curvature Matrices Should Be Democratized via Linear Operators](https://mlsanity.com/paper/2501.19183/) (2025)
- [TENGraD: Time-Efficient Natural Gradient Descent with Exact Fisher-Block Inversion](https://mlsanity.com/paper/2106.03947/) (2021)
- [MKOR: Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 Updates](https://mlsanity.com/paper/2306.01685/) (2023)
- [Kronecker-factored Quasi-Newton Methods for Deep Learning](https://mlsanity.com/paper/2102.06737/) (2021)
- [Rethinking Gauss-Newton for learning over-parameterized models](https://mlsanity.com/paper/2302.02904/) (2023)
- [Theoretical characterisation of the Gauss-Newton conditioning in Neural Networks](https://mlsanity.com/paper/2411.02139/) (2024)
- [Thermodynamic Natural Gradient Descent](https://mlsanity.com/paper/2405.13817/) (2024)
- [Regularized Gauss-Newton for Optimizing Overparameterized Neural Networks](https://mlsanity.com/paper/2404.14875/) (2024)
- [A Riemannian Optimization Perspective of the Gauss-Newton Method for Feedforward Neural Networks](https://mlsanity.com/paper/2412.14031/) (2024)
- [Improving Levenberg-Marquardt Algorithm for Neural Networks](https://mlsanity.com/paper/2212.08769/) (2022)
- [Worth Their Weight: Randomized and Regularized Block Kaczmarz Algorithms without Preprocessing](https://mlsanity.com/paper/2502.00882/) (2025)
- [A Sketch-and-Project Analysis of Subsampled Natural Gradient Algorithms](https://mlsanity.com/paper/2508.21022/) (2025)
- [Rank-1 Approximation of Inverse Fisher for Natural Policy Gradients in Deep Reinforcement Learning](https://mlsanity.com/paper/2601.18626/) (2026)
- [Optimising Distributions with Natural Gradient Surrogates](https://mlsanity.com/paper/2310.11837/) (2023)
- [Incremental Gauss-Newton Descent for Machine Learning](https://mlsanity.com/paper/2408.05560/) (2024)
- [On the Convergence Behavior of Preconditioned Gradient Descent Toward the Rich Learning Regime](https://mlsanity.com/paper/2601.03162/) (2026)
- [Research of Damped Newton Stochastic Gradient Descent Method for Neural Network Training](https://mlsanity.com/paper/2103.16764/) (2021)
- [Dual Gauss-Newton Directions for Deep Learning](https://mlsanity.com/paper/2308.08886/) (2023)
- [Inverse-Free Fast Natural Gradient Descent Method for Deep Learning](https://mlsanity.com/paper/2403.03473/) (2024)
- [Ground state phases of the two-dimension electron gas with a unified variational approach](https://mlsanity.com/paper/2405.19397/) (2024)
- [Second-Order, First-Class: A Composable Stack for Curvature-Aware Training](https://mlsanity.com/paper/2603.25976/) (2026)
- [Fast Gauss-Newton for Multiclass Cross-Entropy](https://mlsanity.com/paper/2605.06081/) (2026)

## Abstract (arXiv)

We present practical Levenberg-Marquardt variants of Gauss-Newton and natural gradient methods for solving non-convex optimization problems that arise in training deep neural networks involving enormous numbers of variables and huge data sets. Our methods use subsampled Gauss-Newton or Fisher information matrices and either subsampled gradient estimates (fully stochastic) or full gradients (semi-stochastic), which, in the latter case, we prove convergent to a stationary point. By using the Sherman-Morrison-Woodbury formula with automatic differentiation (backpropagation) we show how our methods can be implemented to perform efficiently. Finally, numerical results are presented to demonstrate the effectiveness of our proposed methods.