mlsanityData to 14 Oct 2026

Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalization

arXiv 2307.11007 ↗First posted 20 Jul 2023cs.LG math.OC stat.MLcited by 39

Not on the map. Why?

Abstract (arXiv)

Despite extensive studies, the underlying reason as to why overparameterized neural networks can generalize remains elusive. Existing theory shows that common stochastic optimizers prefer flatter minimizers of the training loss, and thus a natural potential explanation is that flatness implies generalization. This work critically examines this explanation. Through theoretical and empirical investigation, we identify the following three scenarios for two-layer ReLU networks: (1) flatness provably implies generalization; (2) there exist non-generalizing flattest models and sharpness minimization algorithms fail to generalize, and (3) perhaps most surprisingly, there exist non-generalizing flattest models, but sharpness minimization algorithms still generalize. Our results suggest that the relationship between sharpness and generalization subtly depends on the data distributions and the model architectures and sharpness minimization algorithms do not only minimize sharpness to achieve better generalization. This calls for the search for other explanations for the generalization of over-parameterized neural networks.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 34 of its 46 references that are in the map. First posted on arXiv 20 Jul 2023.

Builds on

34 of 46 references.

References · 34

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  3. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  5. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  6. 2017Exploring Generalization in Deep Learningcited by 600
  7. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  8. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  9. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  10. 2017Implicit Regularization in Matrix Factorizationcited by 322
  11. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  12. 2017Three Factors Influencing Minima in SGDcited by 273
22 more
  1. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  2. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  3. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  4. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  5. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  6. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 102
  7. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 92
  8. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  9. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  10. 2019Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentationcited by 71
  11. 2019Convergence rates for the stochastic gradient descent method for non-convex objective functionscited by 66
  12. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 65
  13. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 63
  14. 2019Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margincited by 56
  15. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  16. 2023The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descentcited by 44
  17. 2018The loss landscape of overparameterized neural networkscited by 39
  18. 2023Sharpness-Aware Minimization Leads to Low-Rank Featurescited by 38
  19. 2022Flat minima generalize for low-rank matrix recoverycited by 28
  20. 2022Explicit Regularization in Overparametrized Models via Noise Injectioncited by 26
  21. 2019Diametrical Risk Minimization: Theory and Computationscited by 15
  22. 2023The Inductive Bias of Flatness Regularization for Deep Matrix Factorizationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 15; 2025: 13; 2026: 7.

39 papers.

Sort:

By area:

  1. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  2. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  3. 2024A Universal Class of Sharpness-Aware Minimization Algorithmscited by 14
  4. 2024A PAC-Bayesian Link Between Generalisation and Flat Minimacited by 13
  5. 2024Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problemscited by 10
  6. 2026Sharpness-Aware Pretraining Mitigates Catastrophic Forgettingcited by 10
  7. 2024Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learningcited by 9
  8. 2024Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategycited by 8
  9. 2023TRAM: Bridging Trust Regions and Sharpness Aware Minimizationcited by 6
  10. 2024Exploration is Harder than Prediction: Cryptographically Separating Reinforcement Learning from Supervised Learningcited by 6
  11. 2024Tilted Sharpness-Aware Minimizationcited by 6
  12. 2025On Learning Verifiers and Implications to Chain-of-Thought Reasoningcited by 6
27 more
  1. 2025Zeroth-Order Optimization Finds Flat Minimacited by 6
  2. 2026Why Muon Outperforms Adam: A Curvature Perspectivecited by 6
  3. 2023Transferability Bound Theory: Exploring Relationship between Adversarial Transferability and Flatnesscited by 5
  4. 2023Achieving Margin Maximization Exponentially Fast via Progressive Norm Rescalingcited by 4
  5. 2025Flat Minima and Generalization: Insights from Stochastic Convex Optimizationcited by 4
  6. 2024On the Trade-off between Flatness and Optimization in Distributed Learningcited by 3
  7. 2024Towards Understanding the Role of Sharpness-Aware Minimization Algorithms for Out-of-Distribution Generalizationcited by 3
  8. 2024SSE-SAM: Balancing Head and Tail Classes Gradually through Stage-Wise SAMcited by 3
  9. 2025Information-Theoretic Framework for Understanding Modern Machine-Learningcited by 3
  10. 2025Sharp Minima Can Generalize: A Loss Landscape Perspective On Datacited by 3
  11. 2025Towards Understanding The Calibration Benefits of Sharpness-Aware Minimizationcited by 2
  12. 2025Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noisecited by 2
  13. 2025A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Biascited by 2
  14. 2026Generalization at the Edge of Stabilitycited by 2
  15. 2024Improving Generalization of Deep Neural Networks by Optimum Shiftingcited by 1
  16. 2025R'enyi Sharpness: A Novel Sharpness that Strongly Correlates with Generalizationcited by 1
  17. 2025Do Flat Minima Improve Sparse Novel View Synthesis?cited by 1
  18. 2026Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minimacited by 1
  19. 2023A simple connection from loss flatness to compressed neural representationscited by 0
  20. 2024Strong convexity-guided hyper-parameter optimization for flatter lossescited by 0
  21. 2024Forget Sharpness: Perturbed Forgetting of Model Biases Within SAM Dynamicscited by 0
  22. 2025Flatness-Aware Stochastic Gradient Langevin Dynamicscited by 0
  23. 2025A Function-Centric Perspective on Flat and Sharp Minimacited by 0
  24. 2025Zeroth-Order Sharpness-Aware Learning with Exponential Tiltingcited by 0
  25. 2026Why Some Models Resist Unlearning: A Linear Stability Perspectivecited by 0
  26. 2026Certification of Machine Learning Models via Directional Sharpnesscited by 0
  27. 2026On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Boundscited by 0