mlsanityData to 14 Oct 2026

Sharp Minima Can Generalize For Deep Nets

arXiv 1703.04933 ↗First posted 15 Mar 2017cs.LGcited by 466

Not on the map. Why?

Abstract (arXiv)

Despite their overwhelming capacity to overfit, deep learning architectures tend to generalize relatively well to unseen data, allowing them to be deployed in practice. However, explaining why this is the case is still an open area of research. One standing hypothesis that is gaining popularity, e.g. Hochreiter & Schmidhuber (1997); Keskar et al. (2017), is that the flatness of minima of the loss function found by stochastic gradient based methods results in good generalization. This paper argues that most notions of flatness are problematic for deep models and can not be directly applied to explain generalization. Specifically, when focusing on deep networks with rectifier units, we can exploit the particular geometry of parameter space induced by the inherent symmetries that these architectures exhibit to build equivalent models corresponding to arbitrarily sharper minima. Furthermore, if we allow to reparametrize a function, the geometry of its parameters can change drastically without affecting its generalization properties.

Key facts. Cited by 466 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Mar 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 49; 2022: 90; 2023: 71; 2024: 86; 2025: 71; 2026: 98.

466 papers.

Sort:

By area:

  1. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 367
  2. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  3. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  4. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 180
  5. 2023Understanding plasticity in neural networkscited by 174
  6. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  7. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  8. 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
  9. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  10. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  11. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  12. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
38 more
  1. 2023Sharpness-Aware Gradient Matching for Domain Generalizationcited by 102
  2. 2022Sharpness-Aware Training for Freecited by 95
  3. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 92
  4. 2023Deep Model Fusion: A Surveycited by 91
  5. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  6. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
  7. 2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
  8. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  9. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  10. 2022When Do Flat Minima Optimizers Work?cited by 78
  11. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  12. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  13. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 68
  14. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 65
  15. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 63
  16. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  17. 2022Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learningcited by 58
  18. 2023Revisiting Weighted Aggregation in Federated Learning with Neural Networkscited by 58
  19. 2021Relating Adversarially Robust Generalization to Flat Minimacited by 57
  20. 2022Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspectivecited by 57
  21. 2022On the Maximum Hessian Eigenvalue and Generalizationcited by 57
  22. 2023Graph Metanetworks for Processing Diverse Neural Architecturescited by 57
  23. 2022Linear Connectivity Reveals Generalization Strategiescited by 56
  24. 2025Deep Learning is Not So Mysterious or Differentcited by 56
  25. 2024Friendly Sharpness-Aware Minimizationcited by 53
  26. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  27. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 51
  28. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  29. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Modelscited by 46
  30. 2023The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descentcited by 44
  31. 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 43
  32. 2022CoSCL: Cooperation of Small Continual Learners is Stronger than a Big Onecited by 43
  33. 2023An Adaptive Policy to Employ Sharpness-Aware Minimizationcited by 43
  34. 2022Anticorrelated Noise Injection for Improved Generalizationcited by 42
  35. 2024Learning with 3D rotations, a hitchhiker's guide to SO(3)cited by 42
  36. 2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 40
  37. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  38. 2023Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalizationcited by 39

50 most cited of 466 All 466 (JSON)