# Quasi-Newton Quasi-Monte Carlo for variational Bayes

arXiv: https://arxiv.org/abs/2104.02865 · HTML: https://mlsanity.com/paper/2104.02865/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 53 references that are in the map.
- First posted on arXiv 7 Apr 2021.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [Monte Carlo Gradient Estimation in Machine Learning](https://mlsanity.com/paper/1906.10652/) (2019)
- [A Stochastic Quasi-Newton Method for Large-Scale Optimization](https://mlsanity.com/paper/1401.7020/) (2014)
- [A Progressive Batching L-BFGS Method for Machine Learning](https://mlsanity.com/paper/1802.05374/) (2018)
- [A Linearly-Convergent Stochastic L-BFGS Algorithm](https://mlsanity.com/paper/1508.02087/) (2015)
- [A Multi-Batch L-BFGS Method for Machine Learning](https://mlsanity.com/paper/1605.06049/) (2016)
- [Reducing Reparameterization Gradient Variance](https://mlsanity.com/paper/1705.07880/) (2017)
- [Stochastic Block BFGS: Squeezing More Curvature out of Data](https://mlsanity.com/paper/1603.09649/) (2016)
- [Quasi-Monte Carlo Variational Inference](https://mlsanity.com/paper/1807.01604/) (2018)
- [RES: Regularized Stochastic BFGS Algorithm](https://mlsanity.com/paper/1401.7625/) (2014)
- [Scalable inference for crossed random effects models](https://mlsanity.com/paper/1803.09460/) (2018)
- [Backfitting for large scale crossed random effects regressions](https://mlsanity.com/paper/2007.10612/) (2020)
- [Analysis of the BFGS Method with Errors](https://mlsanity.com/paper/1901.09063/) (2019)
- [A Noise-Tolerant Quasi-Newton Algorithm for Unconstrained Optimization](https://mlsanity.com/paper/2010.04352/) (2020)

## Built on by (most cited first, 2 of 2)

- [Federated Variational Inference Methods for Structured Latent Variable Models](https://mlsanity.com/paper/2302.03314/) (2023)
- [Policy Learning and Evaluation with Randomized Quasi-Monte Carlo](https://mlsanity.com/paper/2202.07808/) (2022)

## Abstract (arXiv)

Many machine learning problems optimize an objective that must be measured with noise. The primary method is a first order stochastic gradient descent using one or more Monte Carlo (MC) samples at each step. There are settings where ill-conditioning makes second order methods such as L-BFGS more effective. We study the use of randomized quasi-Monte Carlo (RQMC) sampling for such problems. When MC sampling has a root mean squared error (RMSE) of O(n-1/2) then RQMC has an RMSE of o(n-1/2) that can be close to O(n-3/2) in favorable settings. We prove that improved sampling accuracy translates directly to improved optimization. In our empirical investigations for variational Bayes, using RQMC with stochastic L-BFGS greatly speeds up the optimization, and sometimes finds a better parameter value than MC does.