# Discriminative Bayesian filtering lends momentum to the stochastic Newton method for minimizing log-convex functions

arXiv: https://arxiv.org/abs/2104.12949 · HTML: https://mlsanity.com/paper/2104.12949/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 81 references that are in the map.
- First posted on arXiv 27 Apr 2021.

## Builds on (most cited first, 19 of 19)

- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives](https://mlsanity.com/paper/1407.0202/) (2014)
- [Stochastic Gradient Descent as Approximate Bayesian Inference](https://mlsanity.com/paper/1704.04289/) (2017)
- [Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates](https://mlsanity.com/paper/1905.09997/) (2019)
- [Newton Sketch: A Linear-time Optimization Algorithm with Linear-Quadratic Convergence](https://mlsanity.com/paper/1505.02250/) (2015)
- [Momentum and Stochastic Momentum for Stochastic Gradient, Newton, Proximal Point and Subspace Descent Methods](https://mlsanity.com/paper/1712.09677/) (2017)
- [Exact and Inexact Subsampled Newton Methods for Optimization](https://mlsanity.com/paper/1609.08502/) (2016)
- [Convergence rates of sub-sampled Newton methods](https://mlsanity.com/paper/1508.02810/) (2015)
- [Efficient Second Order Online Learning by Sketching](https://mlsanity.com/paper/1602.02202/) (2016)
- [Adaptive Sampling Strategies for Stochastic Optimization](https://mlsanity.com/paper/1710.11258/) (2017)
- [Probabilistic Line Searches for Stochastic Optimization](https://mlsanity.com/paper/1502.02846/) (2015)
- [Sub-sampled Newton Methods with Non-uniform Sampling](https://mlsanity.com/paper/1607.00559/) (2016)
- [An Investigation of Newton-Sketch and Subsampled Newton Methods](https://mlsanity.com/paper/1705.06211/) (2017)
- [A Subsampling Line-Search Method with Second-Order Results](https://mlsanity.com/paper/1810.07211/) (2018)
- [Particle Filtering Methods for Stochastic Optimization with Application to Large-Scale Empirical Risk Minimization](https://mlsanity.com/paper/1807.08534/) (2018)
- [A Discriminative Approach to Bayesian Filtering with Applications to Human Neural Decoding](https://mlsanity.com/paper/1807.06173/) (2018)
- [Sketching, Embedding, and Dimensionality Reduction for Information Spaces](https://mlsanity.com/paper/1503.05225/) (2015)
- [A probabilistic incremental proximal gradient method](https://mlsanity.com/paper/1812.01655/) (2018)

## Abstract (arXiv)

To minimize the average of a set of log-convex functions, the stochastic Newton method iteratively updates its estimate using subsampled versions of the full objective's gradient and Hessian. We contextualize this optimization problem as sequential Bayesian inference on a latent state-space model with a discriminatively-specified observation process. Applying Bayesian filtering then yields a novel optimization algorithm that considers the entire history of gradients and Hessians when forming an update. We establish matrix-based conditions under which the effect of older observations diminishes over time, in a manner analogous to Polyak's heavy ball momentum. We illustrate various aspects of our approach with an example and review other relevant innovations for the stochastic Newton method.