mlsanityData to 7 Oct 2026

On the Lipschitz Constant of Deep Networks and Double Descent

arXiv 2301.12309 ↗First posted 28 Jan 2023cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Existing bounds on the generalization error of deep networks assume some form of smooth or bounded dependence on the input variable, falling short of investigating the mechanisms controlling such factors in practice. In this work, we present an extensive experimental study of the empirical Lipschitz constant of deep networks undergoing double descent, and highlight non-monotonic trends strongly correlating with the test error. Building a connection between parameter-space and input-space gradients for SGD around a critical point, we isolate two important factors -- namely loss landscape curvature and distance of parameters from initialization -- respectively controlling optimization dynamics around a critical point and bounding model function complexity, even beyond the training data. Our study presents novels insights on implicit regularization via overparameterization, and effective model complexity for networks trained in practice.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 46 of its 61 references that are in the map. First posted on arXiv 28 Jan 2023.

Builds on

46 of 61 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  5. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  6. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  7. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  8. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  9. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  10. 2019Benign Overfitting in Linear Regressioncited by 544
  11. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  12. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
34 more
  1. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  2. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
  3. 2016On the Expressive Power of Deep Neural Networkscited by 302
  4. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  5. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  6. 2016Benefits of depth in neural networkscited by 248
  7. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
  8. 2018Robustness via curvature regularization, and vice versacited by 174
  9. 2019Complexity of Linear Regions in Deep Networkscited by 166
  10. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 160
  11. 2021A Universal Law of Robustness via Isoperimetrycited by 150
  12. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  13. 2020Exactly Computing the Local Lipschitz Constant of ReLU Networkscited by 102
  14. 2018Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolatecited by 99
  15. 2019Are All Layers Created Equal?cited by 94
  16. 2019Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentationcited by 71
  17. 2021On the geometry of generalization and memorization in deep neural networkscited by 71
  18. 2018The jamming transition as a paradigm to understand the loss landscape of deep neural networkscited by 61
  19. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 61
  20. 2022Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspectivecited by 57
  21. 2019Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resiliencecited by 51
  22. 2019On the interplay between noise and curvature and its effect on optimization and generalizationcited by 44
  23. 2019The intriguing role of module criticality in the generalization of deep networkscited by 40
  24. 2021Strength of Minibatch Noise in SGDcited by 39
  25. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 38
  26. 2019How to Initialize your Network? Robust Initialization for WeightNorm & ResNetscited by 36
  27. 2020Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descentcited by 36
  28. 2021Power-law escape rate of SGDcited by 29
  29. 2015An Exploration of Softmax Alternatives Belonging to the Spherical Loss Familycited by 27
  30. 2022Robustness Implies Generalization via Data-Dependent Generalization Boundscited by 23
  31. 2020Extreme Memorization via Scale of Initializationcited by 18
  32. 2022Are All Linear Regions Created Equal?cited by 13
  33. 2022Deep Double Descent via Smooth Interpolationcited by 8
  34. 2019On Second-Moment Stability of Discrete-Time Linear Systems with General Stochastic Dynamicscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 0; 2025: 4; 2026: 3.

11 papers.

  1. 2023Some Fundamental Aspects about Lipschitz Continuity of Neural Networkscited by 25
  2. 2023Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimizationcited by 13
  3. 2023On progressive sharpening, flat minima and generalisationcited by 6
  4. 2025On Space Folds of ReLU Neural Networkscited by 3
  5. 2025On the Complexity-Faithfulness Trade-off of Gradient-Based Explanationscited by 2
  6. 2025On Spectral Properties of Gradient-based Explanation Methodscited by 2
  7. 2023Understanding the Role of Optimization in Double Descentcited by 0
  8. 2025Optimization-Induced Dynamics of Lipschitz Continuity in Neural Networkscited by 0
  9. 2026Deep Exploration of Epoch-wise Double Descent in Noisy Data: Signal Separation, Large Activation, and Benign Overfittingcited by 0
  10. 2026The Price of Robustness: Stable Classifiers Need Overparameterizationcited by 0
  11. 2026Improving Adversarial Robustness of Attribution via Implicit Regularizationcited by 0