How Far Can Sharpness and Complexity Jointly Explain Generalization?
Not on the map. Why?
Abstract (arXiv)
Sharpness and complexity are two central factors in the generalization analysis of deep neural networks. Existing quantitative evaluations of generalization measures have largely focused on individual scalar measures, leaving the joint explanatory power of sharpness and complexity largely unexplored. This work studies how far sharpness and complexity can jointly explain generalization. We use linear regression and introduce a Pareto-based analysis to quantitatively evaluate the joint explanatory power of these two factors. Beyond the existing parameter-level definitions, we further propose realizations of sharpness and complexity that are closer to function space and less dependent on raw parameter representations. We find that function-oriented definitions of these two quantities expand the explanatory scope of the two-factor view beyond what is achieved by existing parameter-level metrics. Overall, our results support the sharpness-complexity perspective as an informative lens for understanding generalization across diverse settings. At the same time, the remaining failures indicate that whether this two-factor view can serve as a complete theory of generalization remains open.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 45 references that are in the map. First posted on arXiv 27 Jun 2026.
Builds on
30 of 45 references.
References · 30
- 2016Understanding deep learning requires rethinking generalizationcited by 1,657
- 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
- 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
- 2017Spectrally-normalized margin bounds for neural networkscited by 681
- 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
- 2017Exploring Generalization in Deep Learningcited by 595
- 2019Benign Overfitting in Linear Regressioncited by 544
- 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
- 2019Fantastic Generalization Measures and Where to Find Themcited by 507
- 2017Sharp Minima Can Generalize For Deep Netscited by 462
18 more
- 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
- 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
- 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
- 2015Norm-Based Capacity Control in Neural Networkscited by 299
- 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
- 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
- 2021User-friendly introduction to PAC-Bayes boundscited by 234
- 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
- 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
- 2019A Primer on PAC-Bayesian Learningcited by 130
- 2015On the properties of variational approximations of Gibbs posteriorscited by 121
- 2018Three Mechanisms of Weight Decay Regularizationcited by 112
- 2022On the Maximum Hessian Eigenvalue and Generalizationcited by 57
- 2020PAC-Bayes unleashed: generalisation bounds with unbounded lossescited by 43
- 2023More PAC-Bayes bounds: From bounded losses, to losses with general tail behaviors, to anytime validitycited by 17
- 2024PAC-Bayes-Chernoff bounds for unbounded lossescited by 9
- 2023Improving Generalization of Complex Models under Unbounded Loss Using PAC-Bayes Boundscited by 6
- 2022Regularizing Deep Neural Networks with Stochastic Estimators of Hessian Tracecited by 4
Built on by
Not cited yet.