# Adaptive Stepsizing for Stochastic Gradient Langevin Dynamics in Bayesian Neural Networks

arXiv: https://arxiv.org/abs/2511.11666 · HTML: https://mlsanity.com/paper/2511.11666/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 25 references that are in the map.
- First posted on arXiv 11 Nov 2025.

## Builds on (most cited first, 11 of 11)

- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis](https://mlsanity.com/paper/1702.03849/) (2017)
- [A Complete Recipe for Stochastic Gradient MCMC](https://mlsanity.com/paper/1506.04696/) (2015)
- [Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks](https://mlsanity.com/paper/1512.07666/) (2015)
- [Bayesian Neural Network Priors Revisited](https://mlsanity.com/paper/2102.06571/) (2021)
- [Stochastic Gradient Langevin Dynamics Algorithms with Adaptive Drifts](https://mlsanity.com/paper/2009.09535/) (2020)
- [Covariance-Controlled Adaptive Langevin Thermostat for Large-Scale Bayesian Sampling](https://mlsanity.com/paper/1510.08692/) (2015)
- [Scalable Stochastic Gradient Riemannian Langevin Dynamics in Non-Diagonal Metrics](https://mlsanity.com/paper/2303.05101/) (2023)
- [Friction-adaptive descent: a family of dynamics-based optimization methods](https://mlsanity.com/paper/2306.06738/) (2023)
- [A Langevin sampling algorithm inspired by the Adam optimizer](https://mlsanity.com/paper/2504.18911/) (2025)
- [On the Convergence of Locally Adaptive and Scalable Diffusion-Based Sampling Methods for Deep Bayesian Neural Network Posteriors](https://mlsanity.com/paper/2403.08609/) (2024)

## Built on by (most cited first, 1 of 1)

- [Accurate Large-sample Uncertainty Quantification using Stochastic Gradient Markov Chain Monte Carlo](https://mlsanity.com/paper/2606.00293/) (2026)

## Abstract (arXiv)

Bayesian neural networks (BNNs) require scalable sampling algorithms to approximate posterior distributions over parameters. Existing stochastic gradient Markov Chain Monte Carlo (SGMCMC) methods are highly sensitive to the choice of stepsize and adaptive variants such as pSGLD typically fail to sample the correct invariant measure without addition of a costly divergence correction term. In this work, we build on the recently proposed `SamAdams' framework for timestep adaptation (Leimkuhler, Lohmann, and Whalley 2025), introducing an adaptive scheme: SA-SGLD, which employs time rescaling to modulate the stepsize according to a monitored quantity (typically the local gradient norm). SA-SGLD can automatically shrink stepsizes in regions of high curvature and expand them in flatter regions, improving both stability and mixing without introducing bias. We show that our method can achieve more accurate posterior sampling than SGLD on high-curvature 2D toy examples and in image classification with BNNs using sharp priors.