# Improving the stability of the covariance-controlled adaptive Langevin thermostat for large-scale Bayesian sampling

arXiv: https://arxiv.org/abs/2512.24515 · HTML: https://mlsanity.com/paper/2512.24515/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 39 references that are in the map.
- First posted on arXiv 30 Dec 2025.

## Builds on (most cited first, 8 of 8)

- [Stochastic Gradient Hamiltonian Monte Carlo](https://mlsanity.com/paper/1402.4102/) (2014)
- [A Complete Recipe for Stochastic Gradient MCMC](https://mlsanity.com/paper/1506.04696/) (2015)
- [Stochastic gradient Markov chain Monte Carlo](https://mlsanity.com/paper/1907.06986/) (2019)
- [On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integrators](https://mlsanity.com/paper/1610.06665/) (2016)
- [Decentralized Stochastic Gradient Langevin Dynamics and Hamiltonian Monte Carlo](https://mlsanity.com/paper/2007.00590/) (2020)
- [Covariance-Controlled Adaptive Langevin Thermostat for Large-Scale Bayesian Sampling](https://mlsanity.com/paper/1510.08692/) (2015)
- [Stochastic Gradient MCMC Methods for Hidden Markov Models](https://mlsanity.com/paper/1706.04632/) (2017)
- [Parameter Expanded Stochastic Gradient Markov Chain Monte Carlo](https://mlsanity.com/paper/2503.00699/) (2025)

## Abstract (arXiv)

Stochastic gradient Langevin dynamics and its variants approximate the likelihood of an entire dataset, via random (and typically much smaller) subsets, in the setting of Bayesian sampling. Due to the (often substantial) improvement of the computational efficiency, they have been widely used in large-scale machine learning applications. It has been demonstrated that the so-called covariance-controlled adaptive Langevin (CCAdL) thermostat, which incorporates an additional term involving the covariance matrix of the noisy force, outperforms popular alternative methods. A moving average is used in CCAdL to estimate the covariance matrix of the noisy force, in which case the covariance matrix will converge to a constant matrix in long-time limit. Moreover, it appears in our numerical experiments that the use of a moving average could reduce the stability of the numerical integrators, thereby limiting the largest usable stepsize. In this article, we propose a modified CCAdL (i.e., mCCAdL) thermostat that uses the scaling part of the scaling and squaring method together with a truncated Taylor series approximation to the exponential to numerically approximate the exact solution to the subsystem involving the additional term proposed in CCAdL. We also propose a symmetric splitting method for mCCAdL, instead of an Euler-type discretisation used in the original CCAdL thermostat. We demonstrate in our numerical experiments that the newly proposed mCCAdL thermostat achieves a substantial improvement in the numerical stability over the original CCAdL thermostat, while significantly outperforming popular alternative stochastic gradient methods in terms of the numerical accuracy for large-scale machine learning applications.