# Inexact Newton Methods for Stochastic Nonconvex Optimization with Applications to Neural Network Training

arXiv: https://arxiv.org/abs/1905.06738 · HTML: https://mlsanity.com/paper/1905.06738/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 May 2019.

## Built on by (most cited first, 4 of 4)

- [Residual-based error correction for neural operator accelerated infinite-dimensional Bayesian inverse problems](https://mlsanity.com/paper/2210.03008/) (2022)
- [slimTrain -- A Stochastic Approximation Method for Training Separable Deep Neural Networks](https://mlsanity.com/paper/2109.14002/) (2021)
- [Neural Nets with a Newton Conjugate Gradient Method on Multiple GPUs](https://mlsanity.com/paper/2208.02017/) (2022)
- [SUPN: Shallow Universal Polynomial Networks for Efficient Function Approximation](https://mlsanity.com/paper/2511.21414/) (2025)

## Abstract (arXiv)

We study stochastic inexact Newton methods and consider their application in nonconvex settings. Building on the work of [R. Bollapragada, R. H. Byrd, and J. Nocedal, IMA Journal of Numerical Analysis, 39 (2018), pp. 545--578] we derive bounds for convergence rates in expected value for stochastic low rank Newton methods, and stochastic inexact Newton Krylov methods. These bounds quantify the errors incurred in subsampling the Hessian and gradient, as well as in approximating the Newton linear solve, and in choosing regularization and step length parameters. We deploy these methods in training convolutional autoencoders for the MNIST and CIFAR10 data sets. Numerical results demonstrate that, relative to first order methods, these stochastic inexact Newton methods often converge faster, are more cost-effective, and generalize better.