mlsanityData to 14 Oct 2026

Tuning Stochastic Gradient Algorithms for Statistical Inference via Large-Sample Asymptotics

arXiv 2207.12395 ↗First posted 25 Jul 2022stat.CO cs.LG stat.MEcited by 4

Not on the map. Why?

Abstract (arXiv)

The tuning of stochastic gradient algorithms (SGAs) for optimization and sampling is often based on heuristics and trial-and-error rather than generalizable theory. We address this theory--practice gap by characterizing the large-sample statistical asymptotics of SGAs via a joint step-size--sample-size scaling limit. We show that iterate averaging with a large fixed step size is robust to the choice of tuning parameters and asymptotically has covariance proportional to that of the MLE sampling distribution. We also prove a Bernstein--von Mises-like theorem to guide tuning, including for generalized posteriors that are robust to model misspecification. Numerical experiments validate our results and recommendations in realistic finite-sample regimes. Our work lays the foundation for a systematic analysis of other stochastic gradient Markov chain Monte Carlo algorithms for a wide range of models.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 25 of its 60 references that are in the map. First posted on arXiv 25 Jul 2022.

Builds on

25 of 60 references.

References · 25

  1. 2019On the Convergence of Adam and Beyondcited by 841
  2. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 343
  3. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
  4. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 243
  5. 2015Non-asymptotic convergence analysis for the Unadjusted Langevin Algorithmcited by 197
  6. 2013A General Framework for Updating Belief Distributionscited by 175
  7. 2015A Complete Recipe for Stochastic Gradient MCMCcited by 146
  8. 2016High-dimensional Bayesian inference via the Unadjusted Langevin Algorithmcited by 146
  9. 2014Analysis and Design of Optimization Algorithms via Integral Quadratic Constraintscited by 105
  10. 2014Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing Itcited by 95
  11. 2017Bridging the Gap between Constant Step Size Stochastic Gradient Descent and Markov Chainscited by 88
  12. 2016Statistical Inference for Model Parameters in Stochastic Gradient Descentcited by 71
13 more
  1. 2014Consistency and fluctuations for stochastic gradient Langevin dynamicscited by 65
  2. 2018The promises and pitfalls of Stochastic Gradient Langevin Dynamicscited by 56
  3. 2019Stochastic gradient Markov chain Monte Carlocited by 56
  4. 2018Quasi-hyperbolic momentum and Adam for deep learningcited by 48
  5. 2017Control Variates for Stochastic Gradient MCMCcited by 25
  6. 2017Statistical inference using SGDcited by 21
  7. 2015The Proximal Robbins-Monro Methodcited by 14
  8. 2017A Robust Accelerated Optimization Algorithm for Strongly Convex Functionscited by 14
  9. 2017The True Cost of Stochastic Gradient Langevin Dynamicscited by 11
  10. 2021Efficient and Generalizable Tuning Strategies for Stochastic Gradient MCMCcited by 11
  11. 2020Reproducible Model Selection Using Bagged Posteriorscited by 7
  12. 2016Quasi-stationary Monte Carlo and the ScaLE Algorithmcited by 5
  13. 2019Robust Inference and Model Criticism Using Bagged Posteriorscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 0; 2026: 2.

4 papers.

  1. 2023Dimension-free mixing times of Gibbs samplers for Bayesian hierarchical modelscited by 4
  2. 2024Enhancing Stochastic Optimization for Statistical Efficiency Using ROOT-SGD with Diminishing Stepsizecited by 0
  3. 2026Accurate Large-sample Uncertainty Quantification using Stochastic Gradient Markov Chain Monte Carlocited by 0
  4. 2026Large-scale Uncertainty Quantification for Latent Variable Models Using Subsampling Markov Chain Monte Carlocited by 0