mlsanityData to 7 Oct 2026

On progressive sharpening, flat minima and generalisation

arXiv 2305.14683 ↗First posted 24 May 2023cs.LG stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

We present a new approach to understanding the relationship between loss curvature and input-output model behaviour in deep learning. Specifically, we use existing empirical analyses of the spectrum of deep network loss Hessians to ground an ansatz tying together the loss Hessian and the input-output Jacobian over training samples during the training of deep neural networks. We then prove a series of theoretical results which quantify the degree to which the input-output Jacobian of a model approximates its Lipschitz norm over a data distribution, and deduce a novel generalisation bound in terms of the empirical Jacobian. We use our ansatz, together with our theoretical results, to give a new account of the recently observed progressive sharpening phenomenon, as well as the generalisation properties of flat minima. Experimental evidence is provided to validate our claims.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 45 of its 67 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

45 of 67 references.

References · 45

  1. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  4. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  5. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  6. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  7. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  9. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  10. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  11. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  12. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
33 more
  1. 2017Exploring Generalization in Deep Learningcited by 595
  2. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  3. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  4. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  5. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  6. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  7. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  8. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  9. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
  10. 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 290
  11. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  12. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  13. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
  14. 2020Stability of Stochastic Gradient Descent on Nonsmooth Convex Lossescited by 172
  15. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  16. 2021A Universal Law of Robustness via Isoperimetrycited by 150
  17. 2019Robust Learning with Jacobian Regularizationcited by 127
  18. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  19. 2017Stability and Generalization of Learning Algorithms that Converge to Global Optimacited by 100
  20. 2017Data-Dependent Stability of Stochastic Gradient Descentcited by 90
  21. 2020Relative Flatness and Generalizationcited by 88
  22. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  23. 2022Understanding the unstable convergence of gradient descentcited by 84
  24. 2019Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentationcited by 71
  25. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  26. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 61
  27. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 61
  28. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 54
  29. 2020Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixturescited by 47
  30. 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
  31. 2020Flatness is a False Friendcited by 19
  32. 2023On the Lipschitz Constant of Deep Networks and Double Descentcited by 11
  33. 2022On skip connections and normalisation layers in deep optimisationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 0; 2026: 1.

6 papers.

  1. 2023Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapultcited by 23
  2. 2024Deep linear networks for regression are implicitly regularized towards flat minimacited by 17
  3. 2023Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimizationcited by 13
  4. 2024Understanding the dynamics of the frequency bias in neural networkscited by 4
  5. 2024Unraveling the Hessian: A Key to Smooth Convergence in Loss Function Landscapescited by 1
  6. 2026Does SGD Seek Flatness or Sharpness? An Exactly Solvable Modelcited by 1