Flatness-Aware Stochastic Gradient Langevin Dynamics
Not on the map. Why?
Abstract (arXiv)
Flatness of the loss landscape has been widely studied as an important perspective for understanding the behavior and generalization of deep learning algorithms. Motivated by this view, we propose Flatness-Aware Stochastic Gradient Langevin Dynamics (fSGLD), a first-order optimization method that biases learning its dynamics toward flat basins while retaining the computational and memory efficiency of SGD and SGLD. We provide a non-asymptotic theoretical analysis showing that fSGLD targets a flatness-biased Gibbs distribution under a theoretically prescribed coupling between the noise scale σ and the inverse temperature β, together with explicit excess risk guarantees. We empirically evaluate fSGLD across standard optimizer benchmarks, Bayesian image classification, uncertainty quantification, and out-of-distribution detection, demonstrating consistently strong performance and reliable uncertainty estimates. Additional experiments confirm the effectiveness of the theoretically prescribed β-σ coupling compared to decoupled choices.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 69 of its 94 references that are in the map. First posted on arXiv 2 Oct 2025.
Builds on
69 of 94 references.
References · 69
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
- 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
- 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 691
- 2017Exploring Generalization in Deep Learningcited by 595
- 2018Predictive Uncertainty Estimation via Prior Networkscited by 555
- 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
- 2019Fantastic Generalization Measures and Where to Find Themcited by 507
- 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
- 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
- 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
38 more
- 2017Three Factors Influencing Minima in SGDcited by 270
- 2021Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotationscited by 253
- 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
- 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
- 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
- 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
- 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
- 2015Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networkscited by 121
- 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
- 2022Selective-Supervised Contrastive Learning with Noisy Labelscited by 112
- 2017Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimizationcited by 96
- 2022Sharpness-Aware Training for Freecited by 95
- 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 91
- 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
- 2020Multi-Objective Interpolation Training for Robustness to Label Noisecited by 86
- 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
- 2024Friendly Sharpness-Aware Minimizationcited by 52
- 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
- 2020A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularizationcited by 44
- 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
- 2023Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalizationcited by 39
- 2019Nonasymptotic estimates for Stochastic Gradient Langevin Dynamics under local conditions in nonconvex optimizationcited by 38
- 2018Non-asymptotic bounds for sampling algorithms without log-concavitycited by 35
- 2020Non-convex Learning via Replica Exchange Stochastic Gradient MCMCcited by 33
- 2022Flat minima generalize for low-rank matrix recoverycited by 28
- 2020A Contour Stochastic Gradient Langevin Dynamics Algorithm for Simulations of Multi-modal Distributionscited by 26
- 2023On diffusion-based generative models and their error bounds: The log-concave case with full convergence estimatescited by 24
- 2020Taming neural networks with TUSLA: Non-convex learning via adaptive stochastic gradient Langevin algorithmscited by 21
- 2024Why is SAM Robust to Label Noise?cited by 21
- 2025Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimizationcited by 21
- 2023Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optimacited by 20
- 2022Improved Convergence Rate of Stochastic Gradient Langevin Dynamics with Variance Reduction and its Application to Optimizationcited by 16
- 2025Wasserstein Convergence of Score-based Generative Models under Semiconvexity and Discontinuous Gradientscited by 15
- 2023Noise Stability Optimization for Finding Flat Minima: A Hessian-based Regularization Approachcited by 14
- 2024Fundamental Convergence Analysis of Sharpness-Aware Minimizationcited by 14
- 2024A Universal Class of Sharpness-Aware Minimization Algorithmscited by 14
- 2021Eliminating Sharp Minima from SGD with Truncated Heavy-tailed Noisecited by 13
- 2023How to escape sharp minima with random perturbationscited by 13
50 most cited of 69 All 69 (JSON)
Built on by
Not cited yet.