Data to 14 Oct 2026

Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks

arXiv 1512.07666 ↗First posted 23 Dec 2015stat.MLcited by 121

Not on the map. Why?

Abstract (arXiv)

Effective training of deep neural networks suffers from two main issues. The first is that the parameter spaces of these models exhibit pathological curvature. Recent methods address this problem by using adaptive preconditioning for Stochastic Gradient Descent (SGD). These methods improve convergence by adapting to the local geometry of parameter space. A second issue is overfitting, which is typically addressed by early stopping. However, recent work has demonstrated that Bayesian model averaging mitigates this problem. The posterior can be sampled by using Stochastic Gradient Langevin Dynamics (SGLD). However, the rapidly changing curvature renders default SGLD methods inefficient. Here, we propose combining adaptive preconditioners with SGLD. In support of this idea, we give theoretical properties on asymptotic convergence and predictive risk. We also provide empirical results for Logistic Regression, Feedforward Neural Nets, and Convolutional Neural Nets, demonstrating that our preconditioned SGLD method gives state-of-the-art performance on these models.

Key facts. Cited by 121 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 Dec 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 15; 2022: 24; 2023: 26; 2024: 18; 2025: 20; 2026: 18.

121 papers.

Sort:

By area:

2021A Survey of Uncertainty in Deep Neural Networkscited by 680
2022Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisonscited by 170
2022A Langevin-like Sampler for Discrete Distributionscited by 44
2023BayesDAG: Gradient-Based Posterior Inference for Causal Discoverycited by 33
2023Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlocited by 29
2023Quantification of Uncertainty with Adversarial Modelscited by 28
2023A Primer on Bayesian Neural Networks: Review and Debatescited by 27
2021Contextual Dropout: An Efficient Sample-Dependent Dropout Modulecited by 26
2022Reliable amortized variational inference with physics-based latent distribution correctioncited by 25
2021Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effectcited by 22
2022Particle-based Variational Inference with Preconditioned Functional Gradient Flowcited by 20
2022Accelerating Score-based Generative Models with Preconditioned Diffusion Samplingcited by 19

38 more

2021Sampling with Mirrored Stein Operatorscited by 17
2024Reparameterization invariance in approximate Bayesian inferencecited by 17
2021When in Doubt: Neural Non-Parametric Uncertainty Quantification for Epidemic Forecastingcited by 16
2021Generalization Bounds for Noisy Iterative Algorithms Using Properties of Additive Noise Channelscited by 15
2025Stochastic Process Learning via Operator Flow Matchingcited by 14
2025Bayesian Influence Functions for Hessian-Free Data Attributioncited by 14
2022How Tempering Fixes Data Augmentation in Bayesian Neural Networkscited by 13
2024Scalable Bayesian Learning with posteriorscited by 13
2021Structured Stochastic Gradient MCMCcited by 12
2021Deep Bayesian inference for seismic imaging with taskscited by 12
2022The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measurecited by 12
2021Efficient and Generalizable Tuning Strategies for Stochastic Gradient MCMCcited by 11
2022Scalable Bayesian Uncertainty Quantification for Neural Network Potentials: Promise and Pitfallscited by 11
2023Tuning-Free Maximum Likelihood Training of Latent Variable Models via Coin Bettingcited by 11
2024Gradient Methods with Online Scalingcited by 11
2022Interacting Contour Stochastic Gradient Langevin Dynamicscited by 9
2023Scalable Stochastic Gradient Riemannian Langevin Dynamics in Non-Diagonal Metricscited by 8
2022Non-reversible Parallel Tempering for Deep Posterior Approximationcited by 7
2022PAC-Bayes Bounds for Bandit Problems: A Survey and Experimental Comparisoncited by 7
2023Inferring networks from time series: a neural approachcited by 7
2024Taming Score-Based Diffusion Priors for Infinite-Dimensional Nonlinear Inverse Problemscited by 7
2024Particle Semi-Implicit Variational Inferencecited by 7
2025Influence Dynamics and Stagewise Data Attributioncited by 7
2026Can Microcanonical Langevin Dynamics Leverage Mini-Batch Gradient Noise?cited by 7
2021Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliabilitycited by 6
2022DiSECt: A Differentiable Simulator for Parameter Inference and Control in Robotic Cuttingcited by 6
2022Effects of Spectral Normalization in Multi-agent Reinforcement Learningcited by 6
2023Preconditioned Score-based Generative Modelscited by 6
2023Fit Like You Sample: Sample-Efficient Generalized Score Matching from Fast Mixing Diffusionscited by 6
2026D-Flow SGLD: Source-Space Posterior Sampling for Scientific Inverse Problems with Flow Matchingcited by 6
2021Batch Normalization Preconditioning for Neural Network Trainingcited by 5
2025From Global to Local: A Scalable Benchmark for Local Posterior Samplingcited by 5
2025Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theorycited by 5
2022PAC-Bayesian Lifelong Learning For Multi-Armed Banditscited by 4
2022A Survey on Uncertainty Toolkits for Deep Learningcited by 4
2023Particle Mean Field Variational Bayescited by 4
2023Non-Log-Concave and Nonsmooth Sampling via Langevin Monte Carlo Algorithmscited by 4
2024NPB-REC: A Non-parametric Bayesian Deep-learning Approach for Undersampled MRI Reconstruction with Uncertainty Estimationcited by 4

50 most cited of 121 All 121 (JSON)