# A Stochastic Quasi-Newton Method with Nesterov's Accelerated Gradient

arXiv: https://arxiv.org/abs/1909.03621 · HTML: https://mlsanity.com/paper/1909.03621/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Sep 2019.

## Built on by (most cited first, 3 of 3)

- [Quasi-Newton Steps for Efficient Online Exp-Concave Optimization](https://mlsanity.com/paper/2211.01357/) (2022)
- [An Adaptive Cost-Sensitive Learning and Recursive Denoising Framework for Imbalanced SVM Classification](https://mlsanity.com/paper/2403.08378/) (2024)
- [Curvature in the Looking-Glass: Optimal Methods to Exploit Curvature of Expectation in the Loss Landscape](https://mlsanity.com/paper/2411.16914/) (2024)

## Abstract (arXiv)

Incorporating second order curvature information in gradient based methods have shown to improve convergence drastically despite its computational intensity. In this paper, we propose a stochastic (online) quasi-Newton method with Nesterov's accelerated gradient in both its full and limited memory forms for solving large scale non-convex optimization problems in neural networks. The performance of the proposed algorithm is evaluated in Tensorflow on benchmark classification and regression problems. The results show improved performance compared to the classical second order oBFGS and oLBFGS methods and popular first order stochastic methods such as SGD and Adam. The performance with different momentum rates and batch sizes have also been illustrated.