# Global Convergence of Online Limited Memory BFGS

arXiv: https://arxiv.org/abs/1409.2045 · HTML: https://mlsanity.com/paper/1409.2045/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Sep 2014.

## Built on by (most cited first, 18 of 18)

- [Machine Unlearning: A Survey](https://mlsanity.com/paper/2306.03558/) (2023)
- [Doubly Adaptive Scaled Algorithm for Machine Learning Using Second-Order Information](https://mlsanity.com/paper/2109.05198/) (2021)
- [Provably Convergent Plug-and-Play Quasi-Newton Methods](https://mlsanity.com/paper/2303.07271/) (2023)
- [SP2: A Second Order Stochastic Polyak Method](https://mlsanity.com/paper/2207.08171/) (2022)
- [Memory Augmented Optimizers for Deep Learning](https://mlsanity.com/paper/2106.10708/) (2021)
- [Optimization for Classical Machine Learning Problems on the GPU](https://mlsanity.com/paper/2203.16340/) (2022)
- [mL-BFGS: A Momentum-based L-BFGS for Distributed Large-Scale Neural Network Optimization](https://mlsanity.com/paper/2307.13744/) (2023)
- [Exploiting Local Convergence of Quasi-Newton Methods Globally: Adaptive Sample Size Approach](https://mlsanity.com/paper/2106.05445/) (2021)
- [Incremental Quasi-Newton Methods with Faster Superlinear Convergence Rates](https://mlsanity.com/paper/2402.02359/) (2024)
- [Newton Losses: Using Curvature Information for Learning with Differentiable Algorithms](https://mlsanity.com/paper/2410.19055/) (2024)
- [A Retrospective Approximation Approach for Smooth Stochastic Optimization](https://mlsanity.com/paper/2103.04392/) (2021)
- [On the efficiency of Stochastic Quasi-Newton Methods for Deep Learning](https://mlsanity.com/paper/2205.09121/) (2022)
- [Sharpened Lazy Incremental Quasi-Newton Method](https://mlsanity.com/paper/2305.17283/) (2023)
- [Shape of Memory: a Geometric Analysis of Machine Unlearning in Second-Order Optimizers](https://mlsanity.com/paper/2604.23046/) (2026)
- [Online Learning Under A Separable Stochastic Approximation Framework](https://mlsanity.com/paper/2305.07484/) (2023)
- [Limited-Memory Greedy Quasi-Newton Method with Non-asymptotic Superlinear Convergence Rate](https://mlsanity.com/paper/2306.15444/) (2023)
- [An Adaptive Cost-Sensitive Learning and Recursive Denoising Framework for Imbalanced SVM Classification](https://mlsanity.com/paper/2403.08378/) (2024)
- [Mo' Memory, Mo' Problems: Stream-Native Machine Unlearning](https://mlsanity.com/paper/2508.10193/) (2025)

## Abstract (arXiv)

Global convergence of an online (stochastic) limited memory version of the Broyden-Fletcher- Goldfarb-Shanno (BFGS) quasi-Newton method for solving optimization problems with stochastic objectives that arise in large scale machine learning is established. Lower and upper bounds on the Hessian eigenvalues of the sample functions are shown to suffice to guarantee that the curvature approximation matrices have bounded determinants and traces, which, in turn, permits establishing convergence to optimal arguments with probability 1. Numerical experiments on support vector machines with synthetic data showcase reductions in convergence time relative to stochastic gradient descent algorithms as well as reductions in storage and computation relative to other online quasi-Newton methods. Experimental evaluation on a search engine advertising problem corroborates that these advantages also manifest in practical applications.