mlsanityData to 7 Oct 2026

A Modern Look at the Relationship between Sharpness and Generalization

arXiv 2302.07011 ↗First posted 14 Feb 2023cs.LGcited by 91

Not on the map. Why?

Abstract (arXiv)

Sharpness of minima is a promising quantity that can correlate with generalization in deep networks and, when optimized during training, can improve generalization. However, standard sharpness is not invariant under reparametrizations of neural networks, and, to fix this, reparametrization-invariant sharpness definitions have been proposed, most prominently adaptive sharpness (Kwon et al., 2021). But does it really capture generalization in modern practical settings? We comprehensively explore this question in a detailed study of various definitions of adaptive sharpness in settings ranging from training from scratch on ImageNet and CIFAR-10 to fine-tuning CLIP on ImageNet and BERT on MNLI. We focus mostly on transformers for which little is known in terms of sharpness despite their widespread usage. Overall, we observe that sharpness does not correlate well with generalization but rather with some training parameters like the learning rate that can be positively or negatively correlated with generalization depending on the setup. Interestingly, in multiple cases, we observe a consistent negative correlation of sharpness with out-of-distribution error implying that sharper minima can generalize better. Finally, we illustrate on a simple model that the right sharpness measure is highly data-dependent, and that we do not understand well this aspect for realistic data distributions. The code of our experiments is available at https://github.com/tml-epfl/sharpness-vs-generalization.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Builds on 57 of its 70 references that are in the map. First posted on arXiv 14 Feb 2023.

Builds on

57 of 70 references.

References · 57

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  8. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  9. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  10. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  11. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  12. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
38 more
  1. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  2. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  3. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  4. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  5. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  6. 2021Robust fine-tuning of zero-shot modelscited by 785
  7. 2017Exploring Generalization in Deep Learningcited by 595
  8. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  9. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  10. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  11. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  12. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  13. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  14. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  15. 2017Three Factors Influencing Minima in SGDcited by 270
  16. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  17. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  18. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  19. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  20. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  21. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  22. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  23. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 125
  24. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  25. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  26. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  27. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  28. 2017Fisher-Rao Metric, Geometry, and Complexity of Neural Networkscited by 99
  29. 2022Sharpness-Aware Training for Freecited by 95
  30. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  31. 2019BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performancecited by 89
  32. 2020Relative Flatness and Generalizationcited by 88
  33. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  34. 2020In Search of Robust Measures of Generalizationcited by 79
  35. 2020Regularizing Neural Networks via Adversarial Model Perturbationcited by 78
  36. 2018A Walk with SGDcited by 74
  37. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  38. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 28; 2025: 25; 2026: 20.

91 papers.

Sort:

By area:

  1. 2024Why are Sensitive Functions Hard for Transformers?cited by 74
  2. 2023Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalizationcited by 39
  3. 2023Normalization Layers Are All That Sharpness-Aware Minimization Needscited by 27
  4. 2023On the Implicit Bias of Adamcited by 25
  5. 2023Improving Convergence and Generalization Using Parameter Symmetriescited by 23
  6. 2023Layer-wise Linear Mode Connectivitycited by 22
  7. 2024Why is SAM Robust to Label Noise?cited by 21
  8. 2024A Survey on Evaluation of Out-of-Distribution Generalizationcited by 20
  9. 2024Deep linear networks for regression are implicitly regularized towards flat minimacited by 17
  10. 2024A Universal Class of Sharpness-Aware Minimization Algorithmscited by 14
  11. 2023How to escape sharp minima with random perturbationscited by 13
  12. 2024Momentum-SAM: Sharpness Aware Minimization without Computational Overheadcited by 13
38 more
  1. 2024A PAC-Bayesian Link Between Generalisation and Flat Minimacited by 13
  2. 2024Transformers Learn Low Sensitivity Functions: Investigations and Implicationscited by 13
  3. 2024QT-DoG: Quantization-aware Training for Domain Generalizationcited by 13
  4. 2025Unveiling the Basin-Like Loss Landscape in Large Language Modelscited by 13
  5. 2023On the different regimes of Stochastic Gradient Descentcited by 12
  6. 2025The Rich and the Simple: On the Implicit Bias of Adam and SGDcited by 12
  7. 2026Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuningcited by 12
  8. 2024Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learningcited by 9
  9. 2024Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategycited by 8
  10. 2025Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokkingcited by 8
  11. 2026Sharpness-Aware Pretraining Mitigates Catastrophic Forgettingcited by 7
  12. 2023The Inductive Bias of Flatness Regularization for Deep Matrix Factorizationcited by 6
  13. 2023TRAM: Bridging Trust Regions and Sharpness Aware Minimizationcited by 6
  14. 2024Where Do Large Learning Rates Lead Us?cited by 6
  15. 2024Tilted Sharpness-Aware Minimizationcited by 6
  16. 2025How Memory in Optimization Algorithms Implicitly Modifies the Losscited by 6
  17. 2023Unveiling the Hessian's Connection to the Decision Boundarycited by 5
  18. 2023Transferability Bound Theory: Exploring Relationship between Adversarial Transferability and Flatnesscited by 5
  19. 2024Theoretical characterisation of the Gauss-Newton conditioning in Neural Networkscited by 5
  20. 2025Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds Itcited by 5
  21. 2025SAFE: Finding Sparse and Flat Minima to Improve Pruningcited by 5
  22. 2025Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and Morecited by 5
  23. 2023Bringing the Discussion of Minima Sharpness to the Audio Domain: a Filter-Normalised Evaluation for Acoustic Scene Classificationcited by 4
  24. 2023A PAC-Bayesian Perspective on the Interpolating Information Criterioncited by 4
  25. 2024A Rescaling-Invariant Lipschitz Bound Based on Path-Metrics for Modern ReLU Network Parameterizationscited by 4
  26. 2026The Effect of Mini-Batch Noise on the Implicit Bias of Adamcited by 4
  27. 2026Cut Less, Fold More: Model Compression through the Lens of Projection Geometrycited by 4
  28. 2026(How) Learning Rates Regulate Catastrophic Overtrainingcited by 4
  29. 2026Why Muon Outperforms Adam: A Curvature Perspectivecited by 4
  30. 2023Stochastic Gradient Descent outperforms Gradient Descent in recovering a high-dimensional signal in a glassy energy landscapecited by 3
  31. 2024MD tree: a model-diagnostic tree grown on loss landscapecited by 3
  32. 2024Sharpness-diversity tradeoff: improving flat ensembles with SharpBalancecited by 3
  33. 2025A Model Zoo on Phase Transitions in Neural Networkscited by 3
  34. 2025When Flatness Does (Not) Guarantee Adversarial Robustnesscited by 3
  35. 2025Flat Minima and Generalization: Insights from Stochastic Convex Optimizationcited by 3
  36. 2025Sharp Minima Can Generalize: A Loss Landscape Perspective On Datacited by 3
  37. 2023Going Further: Flatness at the Rescue of Early Stopping for Adversarial Example Transferabilitycited by 2
  38. 2023Using Stochastic Gradient Descent to Smooth Nonconvex Functions: Analysis of Implicit Graduated Optimizationcited by 2

50 most cited of 91 All 91 (JSON)