mlsanityData to 7 Oct 2026

The Geometric Occam's Razor Implicit in Deep Learning

arXiv 2111.15090 ↗First posted 30 Nov 2021cs.LG stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

In over-parameterized deep neural networks there can be many possible parameter configurations that fit the training data exactly. However, the properties of these interpolating solutions are poorly understood. We argue that over-parameterized neural networks trained with stochastic gradient descent are subject to a Geometric Occam's Razor; that is, these networks are implicitly regularized by the geometric model complexity. For one-dimensional regression, the geometric model complexity is simply given by the arc length of the function. For higher-dimensional settings, the geometric model complexity depends on the Dirichlet energy of the function. We explore the relationship between this Geometric Occam's Razor, the Dirichlet energy and other known forms of implicit regularization. Finally, for ResNets trained on CIFAR-10, we observe that Dirichlet energy measurements are consistent with the action of this implicit Geometric Occam's Razor.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 22 of its 34 references that are in the map. First posted on arXiv 30 Nov 2021.

Builds on

22 of 34 references.

References · 22

  1. 2017Improved Training of Wasserstein GANscited by 2,223
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  4. 2018Which Training Methods for GANs do actually Converge?cited by 672
  5. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  6. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 253
  7. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  8. 2020Implicit Gradient Regularizationcited by 145
  9. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  10. 2019Robust Learning with Jacobian Regularizationcited by 127
  11. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  12. 2018The Mechanics of n-Player Differentiable Gamescited by 118
10 more
  1. 2018Large Margin Deep Networks for Classificationcited by 110
  2. 2017The Numerics of GANscited by 104
  3. 2017Gradient descent GAN optimization is locally stablecited by 65
  4. 2017Many Paths to Equilibrium: GANs Do Not Need to Decrease a Divergence At Every Stepcited by 45
  5. 2020A case for new neural network smoothness constraintscited by 43
  6. 2018On gradient regularizers for MMD GANscited by 28
  7. 2017Gradient Regularization Improves Accuracy of Discriminative Modelscited by 26
  8. 2020Why Spectral Normalization Stabilizes GANs: Analysis and Improvementscited by 21
  9. 2020Training Generative Adversarial Networks by Solving Ordinary Differential Equationscited by 21
  10. 2016Degrees of Freedom in Deep Neural Networkscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 1; 2024: 2; 2025: 1; 2026: 1.

7 papers.

  1. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 38
  2. 2023Implicit biases in multitask and continual learning from a backward error analysis perspectivecited by 4
  3. 2024The Impact of Geometric Complexity on Neural Collapse in Transfer Learningcited by 4
  4. 2022Manifold Characteristics That Predict Downstream Task Performancecited by 2
  5. 2024A Margin-based Multiclass Generalization Bound via Geometric Complexitycited by 2
  6. 2025Polyra Swarms: A Shape-Based Approach to Machine Learningcited by 0
  7. 2026Penalizing Localized Dirichlet Energies in Low Rank Tensor Productscited by 0