# Stochastic quasi-Newton with line-search regularization

arXiv: https://arxiv.org/abs/1909.01238 · HTML: https://mlsanity.com/paper/1909.01238/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Sep 2019.

## Built on by (most cited first, 6 of 6)

- [High-Dimensional Gaussian Process Inference with Derivatives](https://mlsanity.com/paper/2102.07542/) (2021)
- [Exact Gauss-Newton Optimization for Training Deep Neural Networks](https://mlsanity.com/paper/2405.14402/) (2024)
- [Efficient Learning of the Parameters of Non-Linear Models using Differentiable Resampling in Particle Filters](https://mlsanity.com/paper/2111.01409/) (2021)
- [A Novel Fast Exact Subproblem Solver for Stochastic Quasi-Newton Cubic Regularized Optimization](https://mlsanity.com/paper/2204.09116/) (2022)
- [GP-FL: Model-Based Hessian Estimation for Second-Order Over-the-Air Federated Learning](https://mlsanity.com/paper/2412.03867/) (2024)
- [Second-Order, First-Class: A Composable Stack for Curvature-Aware Training](https://mlsanity.com/paper/2603.25976/) (2026)

## Abstract (arXiv)

In this paper we present a novel quasi-Newton algorithm for use in stochastic optimisation. Quasi-Newton methods have had an enormous impact on deterministic optimisation problems because they afford rapid convergence and computationally attractive algorithms. In essence, this is achieved by learning the second-order (Hessian) information based on observing first-order gradients. We extend these ideas to the stochastic setting by employing a highly flexible model for the Hessian and infer its value based on observing noisy gradients. In addition, we propose a stochastic counterpart to standard line-search procedures and demonstrate the utility of this combination on maximum likelihood identification for general nonlinear state space models.