# Stochastic Quasi-Newton Langevin Monte Carlo

arXiv: https://arxiv.org/abs/1602.03442 · HTML: https://mlsanity.com/paper/1602.03442/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Feb 2016.

## Built on by (most cited first, 14 of 14)

- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [SNIPS: Solving Noisy Inverse Problems Stochastically](https://mlsanity.com/paper/2105.14951/) (2021)
- [Sampling with Riemannian Hamiltonian Monte Carlo in a Constrained Space](https://mlsanity.com/paper/2202.01908/) (2022)
- [Sampling with Mirrored Stein Operators](https://mlsanity.com/paper/2106.12506/) (2021)
- [Constrained Exploration via Reflected Replica Exchange Stochastic Gradient Langevin Dynamics](https://mlsanity.com/paper/2405.07839/) (2024)
- [Fit Like You Sample: Sample-Efficient Generalized Score Matching from Fast Mixing Diffusions](https://mlsanity.com/paper/2306.09332/) (2023)
- [Batch Normalization Preconditioning for Neural Network Training](https://mlsanity.com/paper/2108.01110/) (2021)
- [Unleashing High-Quality Image Generation in Diffusion Sampling Using Second-Order Levenberg-Marquardt-Langevin](https://mlsanity.com/paper/2505.24222/) (2025)
- [A stochastic Stein Variational Newton method](https://mlsanity.com/paper/2204.09039/) (2022)
- [New affine invariant ensemble samplers and their dimensional scaling](https://mlsanity.com/paper/2505.02987/) (2025)
- [Learning Rate Free Sampling in Constrained Domains](https://mlsanity.com/paper/2305.14943/) (2023)
- [Langevin algorithms for very deep Neural Networks with application to image classification](https://mlsanity.com/paper/2212.14718/) (2022)
- [Exploring Non-Convex Discrete Energy Landscapes: An Efficient Langevin-Like Sampler with Replica Exchange](https://mlsanity.com/paper/2501.17323/) (2025)
- [Time-Inhomogeneous Preconditioned Langevin Dynamics](https://mlsanity.com/paper/2605.06091/) (2026)

## Abstract (arXiv)

Recently, Stochastic Gradient Markov Chain Monte Carlo (SG-MCMC) methods have been proposed for scaling up Monte Carlo computations to large data problems. Whilst these approaches have proven useful in many applications, vanilla SG-MCMC might suffer from poor mixing rates when random variables exhibit strong couplings under the target densities or big scale differences. In this study, we propose a novel SG-MCMC method that takes the local geometry into account by using ideas from Quasi-Newton optimization methods. These second order methods directly approximate the inverse Hessian by using a limited history of samples and their gradients. Our method uses dense approximations of the inverse Hessian while keeping the time and memory complexities linear with the dimension of the problem. We provide a formal theoretical analysis where we show that the proposed method is asymptotically unbiased and consistent with the posterior expectations. We illustrate the effectiveness of the approach on both synthetic and real datasets. Our experiments on two challenging applications show that our method achieves fast convergence rates similar to Riemannian approaches while at the same time having low computational requirements similar to diagonal preconditioning approaches.