# Comparing BFGS and OGR for Second-Order Optimization

arXiv: https://arxiv.org/abs/2512.06969 · HTML: https://mlsanity.com/paper/2512.06969/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 11 references that are in the map.
- First posted on arXiv 7 Dec 2025.

## Builds on (most cited first, 6 of 6)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Quasi-Newton Methods: Superlinear Convergence Without Line Searches for Self-Concordant Functions](https://mlsanity.com/paper/1612.06965/) (2016)
- [Non-asymptotic Global Convergence Analysis of BFGS with the Armijo-Wolfe Line Search](https://mlsanity.com/paper/2404.16731/) (2024)
- [SGD momentum optimizer with step estimation by online parabola model](https://mlsanity.com/paper/1907.07063/) (2019)
- [Improving SGD convergence by online linear regression of gradients in multiple statistically relevant directions](https://mlsanity.com/paper/1901.11457/) (2019)
- [Statistical and Computational Complexities of BFGS Quasi-Newton Method for Generalized Linear Models](https://mlsanity.com/paper/2206.00207/) (2022)

## Abstract (arXiv)

Estimating the Hessian matrix, especially for neural network training, is a challenging problem due to high dimensionality and cost. In this work, we compare the classical Sherman-Morrison update used in the popular BFGS method (Broy-den-Fletcher-Goldfarb-Shanno), which maintains a positive definite Hessian approximation under a convexity assumption, with a novel approach called Online Gradient Regression (OGR). OGR performs regression of gradients against positions using an exponential moving average to estimate second derivatives online, without requiring Hessian inversion. Unlike BFGS, OGR allows estimation of a general (not necessarily positive definite) Hessian and can thus handle non-convex structures. We evaluate both methods across standard test functions and demonstrate that OGR achieves faster convergence and improved loss, particularly in non-convex settings.