# Training Neural Networks with Stochastic Hessian-Free Optimization

arXiv: https://arxiv.org/abs/1301.3641 · HTML: https://mlsanity.com/paper/1301.3641/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jan 2013.

## Built on by (most cited first, 9 of 9)

- [Federated Learning and Meta Learning: Approaches, Applications, and Directions](https://mlsanity.com/paper/2210.13111/) (2022)
- [Exact Gauss-Newton Optimization for Training Deep Neural Networks](https://mlsanity.com/paper/2405.14402/) (2024)
- [SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation](https://mlsanity.com/paper/2502.18153/) (2025)
- [Incremental Gauss-Newton Descent for Machine Learning](https://mlsanity.com/paper/2408.05560/) (2024)
- [Newton methods based convolution neural networks using parallel processing](https://mlsanity.com/paper/2112.01401/) (2021)
- [Training Autoencoders Using Stochastic Hessian-Free Optimization with LSMR](https://mlsanity.com/paper/2504.13302/) (2025)
- [Stable Coresets via Posterior Sampling: Aligning Induced and Full Loss Landscapes](https://mlsanity.com/paper/2511.17399/) (2025)
- [Second-Order, First-Class: A Composable Stack for Curvature-Aware Training](https://mlsanity.com/paper/2603.25976/) (2026)
- [Fast Gauss-Newton for Multiclass Cross-Entropy](https://mlsanity.com/paper/2605.06081/) (2026)

## Abstract (arXiv)

Hessian-free (HF) optimization has been successfully used for training deep autoencoders and recurrent networks. HF uses the conjugate gradient algorithm to construct update directions through curvature-vector products that can be computed on the same order of time as gradients. In this paper we exploit this property and study stochastic HF with gradient and curvature mini-batches independent of the dataset size. We modify Martens' HF for these settings and integrate dropout, a method for preventing co-adaptation of feature detectors, to guard against overfitting. Stochastic Hessian-free optimization gives an intermediary between SGD and HF that achieves competitive performance on both classification and deep autoencoder experiments.