mlsanityData to 14 Oct 2026

Scalable Stochastic Gradient Riemannian Langevin Dynamics in Non-Diagonal Metrics

arXiv 2303.05101 ↗First posted 9 Mar 2023cs.LG stat.COcited by 8

Not on the map. Why?

Abstract (arXiv)

Stochastic-gradient sampling methods are often used to perform Bayesian inference on neural networks. It has been observed that the methods in which notions of differential geometry are included tend to have better performances, with the Riemannian metric improving posterior exploration by accounting for the local curvature. However, the existing methods often resort to simple diagonal metrics to remain computationally efficient. This loses some of the gains. We propose two non-diagonal metrics that can be used in stochastic-gradient samplers to improve convergence and exploration but have only a minor computational overhead over diagonal metrics. We show that for fully connected neural networks (NNs) with sparsity-inducing priors and convolutional NNs with correlated priors, using these metrics can provide improvements. For some other choices the posterior is sufficiently easy also for the simpler metrics.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 34 of its 57 references that are in the map. First posted on arXiv 9 Mar 2023.

Builds on

34 of 57 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  4. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  5. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  6. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  7. 2015Weight Uncertainty in Neural Networkscited by 554
  8. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  9. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  10. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 343
  11. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 217
  12. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
22 more
  1. 2015A Complete Recipe for Stochastic Gradient MCMCcited by 146
  2. 2020Uncertainty Quantification and Deep Ensemblescited by 139
  3. 2015Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networkscited by 121
  4. 2021Bayesian Neural Network Priors Revisitedcited by 121
  5. 2020All You Need is a Good Functional Prior for Bayesian Deep Learningcited by 61
  6. 2017Model Selection in Bayesian Neural Networks via Horseshoe Priorscited by 52
  7. 2022On Uncertainty, Tempering, and Data Augmentation in Bayesian Classificationcited by 46
  8. 2015Bayesian Dark Knowledgecited by 45
  9. 2019Challenges in Markov chain Monte Carlo for Bayesian neural networkscited by 44
  10. 2016On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integratorscited by 35
  11. 2019Escaping Saddle Points with Adaptive Gradient Methodscited by 34
  12. 2019Benchmarking the Neural Linear Model for Regressioncited by 31
  13. 2021QLSD: Quantised Langevin stochastic dynamics for Bayesian federated learningcited by 26
  14. 2020Scalable Second Order Optimization for Deep Learningcited by 21
  15. 2021BNNpriors: A library for Bayesian neural network inference with different prior distributionscited by 20
  16. 2022Efficiently Training Low-Curvature Neural Networkscited by 19
  17. 2020Stochastic Gradient Langevin Dynamics Algorithms with Adaptive Driftscited by 12
  18. 2022Lagrangian Manifold Monte Carlo on Monge Patchescited by 11
  19. 2022Variational Inference for Infinitely Deep Neural Networkscited by 10
  20. 2017Natural Langevin Dynamics for Neural Networkscited by 6
  21. 2019Interpretable Outcome Prediction with Sparse Bayesian Neural Networks in Intensive Carecited by 6
  22. 2022Variational Inference of overparameterized Bayesian Neural Networks: a theoretical and empirical studycited by 5

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 2; 2025: 4.

8 papers.

  1. 2023Riemannian Laplace Approximation with the Fisher Metriccited by 7
  2. 2025What's Inside Your Diffusion Model? A Score-Based Riemannian Metric to Explore the Data Manifoldcited by 6
  3. 2025Devil is in the Details: Density Guidance for Detail-Aware Generation with Flow Modelscited by 5
  4. 2023Warped geometric information on the optimisation of Euclidean functionscited by 3
  5. 2024On the Convergence of Locally Adaptive and Scalable Diffusion-Based Sampling Methods for Deep Bayesian Neural Network Posteriorscited by 1
  6. 2025Adaptive Stepsizing for Stochastic Gradient Langevin Dynamics in Bayesian Neural Networkscited by 1
  7. 2024Subspace Langevin Monte Carlocited by 0
  8. 2025Feature Preserving Shrinkage on Bayesian Neural Networks via the R2D2 Priorcited by 0