mlsanityData to 7 Oct 2026

Why neural networks find simple solutions: the many regularizers of geometric complexity

arXiv 2209.13083 ↗First posted 27 Sep 2022cs.LG stat.MLcited by 38

Not on the map. Why?

Abstract (arXiv)

In many contexts, simpler models are preferable to more complex models and the control of this model complexity is the goal for many methods in machine learning such as regularization, hyperparameter tuning and architecture design. In deep learning, it has been difficult to understand the underlying mechanisms of complexity control, since many traditional measures are not naturally suitable for deep neural networks. Here we develop the notion of geometric complexity, which is a measure of the variability of the model function, computed using a discrete Dirichlet energy. Using a combination of theoretical arguments and empirical results, we show that many common training heuristics such as parameter norm regularization, spectral norm regularization, flatness regularization, implicit gradient regularization, noise regularization and the choice of parameter initialization all act to control geometric complexity, providing a unifying framework in which to characterize the behavior of deep learning models.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 62 of its 107 references that are in the map. First posted on arXiv 27 Sep 2022.

Builds on

62 of 107 references.

References · 62

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  2. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  3. 2017Improved Training of Wasserstein GANscited by 2,223
  4. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  5. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  6. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  7. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  9. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  10. 2016Geometric deep learning: going beyond Euclidean datacited by 1,042
  11. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  12. 2018Self-Attention Generative Adversarial Networkscited by 809
38 more
  1. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  2. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  3. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  4. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  5. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  6. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  7. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 304
  8. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  9. 2018An Empirical Model of Large-Batch Trainingcited by 268
  10. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  11. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  12. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 253
  13. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 213
  14. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
  15. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  16. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 178
  17. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
  18. 2021A Universal Law of Robustness via Isoperimetrycited by 150
  19. 2019Adversarial Robustness through Local Linearizationcited by 146
  20. 2020Implicit Gradient Regularizationcited by 145
  21. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  22. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  23. 2019Robust Learning with Jacobian Regularizationcited by 127
  24. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  25. 2018The Mechanics of n-Player Differentiable Gamescited by 118
  26. 2016Robust Large Margin Deep Neural Networkscited by 117
  27. 2018Predicting the Generalization Gap in Deep Networks with Margin Distributionscited by 111
  28. 2018Large Margin Deep Networks for Classificationcited by 110
  29. 2017The Numerics of GANscited by 104
  30. 2016Why does deep and cheap learning work so well?cited by 100
  31. 2017Implicit Regularization in Deep Learningcited by 90
  32. 2020In Search of Robust Measures of Generalizationcited by 79
  33. 2017Gradient descent GAN optimization is locally stablecited by 65
  34. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
  35. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  36. 2020Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisitedcited by 50
  37. 2019Generalization in Deep Networks: The Role of Distance from Initializationcited by 49
  38. 2017Many Paths to Equilibrium: GANs Do Not Need to Decrease a Divergence At Every Stepcited by 45

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 11; 2024: 8; 2025: 11; 2026: 7.

38 papers.

Sort:

By area:

  1. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 50
  2. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  3. 2025Learning without training: The implicit dynamics of in-context learningcited by 41
  4. 2023Why Does Little Robustness Help? A Further Step Towards Understanding Adversarial Transferabilitycited by 32
  5. 2023Some Fundamental Aspects about Lipschitz Continuity of Neural Networkscited by 25
  6. 2023Sparse Modular Activation for Efficient Sequence Modelingcited by 20
  7. 2025Training Transformers with Enforced Lipschitz Constantscited by 18
  8. 2024The Pitfalls of Memorization: When Memorization Hurts Generalizationcited by 16
  9. 2025Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wildcited by 13
  10. 2023On the Lipschitz Constant of Deep Networks and Double Descentcited by 11
  11. 2025Interpretable Machine Learning in Physics: A Reviewcited by 11
  12. 2022Relating Regularization and Generalization through the Intrinsic Dimension of Activationscited by 9
26 more
  1. 2023Understanding Pathologies of Deep Heteroskedastic Regressioncited by 9
  2. 2024On the Local Complexity of Linear Regions in Deep ReLU Networkscited by 9
  3. 2023Per-Example Gradient Regularization Improves Learning Signals from Noisy Datacited by 7
  4. 2023Implicit biases in multitask and continual learning from a backward error analysis perspectivecited by 4
  5. 2023Adversarial Estimation of Topological Dimension with Harmonic Score Mapscited by 4
  6. 2024The Impact of Geometric Complexity on Neural Collapse in Transfer Learningcited by 4
  7. 2024A Bayesian Model Selection Criterion for Selecting Pretraining Checkpointscited by 4
  8. 2023Depth Degeneracy in Neural Networks: Vanishing Angles in Fully Connected ReLU Networks on Initializationcited by 3
  9. 2024Disentangling Latent Shifts of In-Context Learning with Weak Supervisioncited by 3
  10. 2025Context Parametrization with Compositional Adapterscited by 3
  11. 2026Stagewise Reinforcement Learning and the Geometry of the Regret Landscapecited by 3
  12. 2026Spectral methods: crucial for machine learning, natural for quantum computers?cited by 3
  13. 2024A Margin-based Multiclass Generalization Bound via Geometric Complexitycited by 2
  14. 2025Large Learning Rates Simultaneously Achieve Robustness to Spurious Correlations and Compressibilitycited by 2
  15. 2025Training Deep Physics-Informed Kolmogorov-Arnold Networkscited by 2
  16. 2024From Robustness to Improved Generalization and Calibration in Pre-trained Language Modelscited by 1
  17. 2025Learning by solving differential equationscited by 1
  18. 2025On residual network depthcited by 1
  19. 2026Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledgecited by 1
  20. 2023Mathematical Challenges in Deep Learningcited by 0
  21. 2025Optimization Landscapes Learned: Proxy Networks Boost Convergence in Physics-based Inverse Problemscited by 0
  22. 2025On the Effect of Regularization on Nonparametric Mean-Variance Regressioncited by 0
  23. 2026Dreaming Smoothly and Sample Efficiently with Gradient Penalized Latent Dynamicscited by 0
  24. 2026Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Biascited by 0
  25. 2026Improving Adversarial Robustness of Attribution via Implicit Regularizationcited by 0
  26. 2026An adaptive framework for the axisymmetric pulsar magnetosphere using physics-informed Kolmogorov-Arnold networkscited by 0