mlsanityData to 14 Oct 2026

From Global to Local: A Scalable Benchmark for Local Posterior Sampling

arXiv 2507.21449 ↗First posted 29 Jul 2025stat.ML cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Degeneracy is an inherent feature of the loss landscape of neural networks, but it is not well understood how stochastic gradient MCMC (SGMCMC) algorithms interact with this degeneracy. In particular, existing global convergence guarantees for common SGMCMC algorithms rely on assumptions which are likely incompatible with degenerate loss landscapes. In this paper, we argue that this gap requires a shift in focus from global to local posterior sampling, and, as a first step, we introduce a novel scalable benchmark for evaluating the local sampling performance of SGMCMC algorithms. We evaluate a number of common algorithms, and find that RMSProp-preconditioned SGLD is most effective at faithfully representing the local geometry of the posterior distribution among the samplers we evaluate. Although we lack theoretical guarantees about global sampler convergence, our empirical results show that we are able to extract non-trivial local information in models with up to O(100M) parameters.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 36 of its 74 references that are in the map. First posted on arXiv 29 Jul 2025.

Builds on

36 of 74 references.

References · 36

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  4. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  5. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  6. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 343
  7. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  8. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 243
  9. 2015Non-asymptotic convergence analysis for the Unadjusted Langevin Algorithmcited by 197
  10. 2015A Complete Recipe for Stochastic Gradient MCMCcited by 146
  11. 2017L2 Regularization versus Batch and Weight Normalizationcited by 124
  12. 2015Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networkscited by 121
24 more
  1. 2017User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradientcited by 116
  2. 2017Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimizationcited by 97
  3. 2020Shortformer: Better Language Modeling using Shorter Inputscited by 76
  4. 2017A Hitting Time Analysis of Stochastic Gradient Langevin Dynamicscited by 71
  5. 2018Sharp convergence rates for Langevin dynamics in the nonconvex settingcited by 68
  6. 2014Consistency and fluctuations for stochastic gradient Langevin dynamicscited by 65
  7. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 62
  8. 2019Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysiscited by 59
  9. 2018The promises and pitfalls of Stochastic Gradient Langevin Dynamicscited by 56
  10. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 51
  11. 2020Deep Learning is Singular, and That's Goodcited by 49
  12. 2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 40
  13. 2019Nonasymptotic estimates for Stochastic Gradient Langevin Dynamics under local conditions in nonconvex optimizationcited by 38
  14. 2016On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integratorscited by 35
  15. 2020Faster Convergence of Stochastic Gradient Langevin Dynamics for Non-Log-Concave Samplingcited by 32
  16. 2017The Tamed Unadjusted Langevin Algorithmcited by 31
  17. 2023Dynamical versus Bayesian Phase Transitions in a Toy Model of Superpositioncited by 29
  18. 2025Dynamics of Transient Structure in In-Context Linear Regression Transformerscited by 14
  19. 2025Structural Inference: Interpreting Small Language Models with Susceptibilitiescited by 13
  20. 2020Stochastic Gradient Langevin Dynamics Algorithms with Adaptive Driftscited by 12
  21. 2019Positively Scale-Invariant Flatness of ReLU Neural Networkscited by 11
  22. 2018Local Optimality and Generalization Guarantees for the Langevin Algorithm via Empirical Metastabilitycited by 6
  23. 2024Geometry of fibers of the multiplication map of deep linear neural networkscited by 4
  24. 2015(Non-) asymptotic properties of Stochastic Gradient Langevin Dynamicscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025Bayesian Influence Functions for Hessian-Free Data Attributioncited by 14
  2. 2026Stagewise Reinforcement Learning and the Geometry of the Regret Landscapecited by 4
  3. 2026Linear Response Estimators for Singular Statistical Modelscited by 1
  4. 2026Exact Algebraic Computation of Learning Coefficients for Two-Dimensional Singular Modelscited by 0
  5. 2026Patterning in Practice: Debiasing Reward Models with Susceptibilitiescited by 0