Data to 14 Oct 2026

Entropy-SGD: Biasing Gradient Descent Into Wide Valleys

arXiv 1611.01838 ↗First posted 6 Nov 2016cs.LG stat.MLcited by 359

Not on the map. Why?

Abstract (arXiv)

This paper proposes a new optimization algorithm called Entropy-SGD for training deep neural networks that is motivated by the local geometry of the energy landscape. Local extrema with low generalization error have a large proportion of almost-zero eigenvalues in the Hessian with very few positive or negative eigenvalues. We leverage upon this observation to construct a local-entropy-based objective function that favors well-generalizable solutions lying in large flat regions of the energy landscape, while avoiding poorly-generalizable solutions located in the sharp valleys. Conceptually, our algorithm resembles two nested loops of SGD where we use Langevin dynamics in the inner loop to compute the gradient of the local entropy before each update of the weights. We show that the new objective has a smoother energy landscape and show improved generalization over SGD using uniform stability, under certain assumptions. Our experiments on convolutional and recurrent networks demonstrate that Entropy-SGD compares favorably to state-of-the-art techniques in terms of generalization error and training time.

Key facts. Cited by 359 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 61; 2022: 65; 2023: 67; 2024: 63; 2025: 55; 2026: 48.

359 papers.

Sort:

By area:

2021SWAD: Domain Generalization by Seeking Flat Minimacited by 434
2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 367
2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
2022Class-Incremental Continual Learning into the eXtended DER-versecited by 124
2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122

38 more

2022Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning Systemcited by 112
2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
2021Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learningcited by 101
2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 92
2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
2021Learning Neural Network Subspacescited by 82
2022When Do Flat Minima Optimizers Work?cited by 78
2022PAC-Bayes Compression Bounds So Tight That They Can Explain Generalizationcited by 76
2023Enhancing Fine-Tuning Based Backdoor Defense with Sharpness-Aware Minimizationcited by 64
2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 63
2021Annealing Knowledge Distillationcited by 61
2021Relating Adversarially Robust Generalization to Flat Minimacited by 57
2022Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspectivecited by 57
2022On the Maximum Hessian Eigenvalue and Generalizationcited by 57
2023Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalizationcited by 57
2024Friendly Sharpness-Aware Minimizationcited by 53
2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 51
2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
2021RDA: Robust Domain Adaptation via Fourier Adversarial Attackingcited by 48
2024Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitatecited by 47
2021A study on the plasticity of neural networkscited by 44
2024SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attentioncited by 44
2023Rethinking Confidence Calibration for Failure Predictioncited by 43
2023An Adaptive Policy to Employ Sharpness-Aware Minimizationcited by 43
2022Anticorrelated Noise Injection for Improved Generalizationcited by 42
2022SAM as an Optimal Relaxation of Bayescited by 41
2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
2021Stability of SGD: Tightness Analysis and Improved Boundscited by 33
2021Hessian Eigenspectra of More Realistic Nonlinear Modelscited by 33
2024Out-of-Domain Generalization in Dynamical Systems Reconstructioncited by 33
2021On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implicationscited by 31
2022A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biasescited by 30
2022Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task modelscited by 29

50 most cited of 359 All 359 (JSON)