# The Geometric Occam's Razor Implicit in Deep Learning

arXiv: https://arxiv.org/abs/2111.15090 · HTML: https://mlsanity.com/paper/2111.15090/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 30 Nov 2021.

## Builds on (most cited first, 22 of 22)

- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Which Training Methods for GANs do actually Converge?](https://mlsanity.com/paper/1801.04406/) (2018)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Understanding Deep Neural Networks with Rectified Linear Units](https://mlsanity.com/paper/1611.01491/) (2016)
- [On the Origin of Implicit Regularization in Stochastic Gradient Descent](https://mlsanity.com/paper/2101.12176/) (2021)
- [Implicit Gradient Regularization](https://mlsanity.com/paper/2009.11162/) (2020)
- [Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation](https://mlsanity.com/paper/2105.14368/) (2021)
- [Robust Learning with Jacobian Regularization](https://mlsanity.com/paper/1908.02729/) (2019)
- [Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like process](https://mlsanity.com/paper/1904.09080/) (2019)
- [The Mechanics of n-Player Differentiable Games](https://mlsanity.com/paper/1802.05642/) (2018)
- [Large Margin Deep Networks for Classification](https://mlsanity.com/paper/1803.05598/) (2018)
- [The Numerics of GANs](https://mlsanity.com/paper/1705.10461/) (2017)
- [Gradient descent GAN optimization is locally stable](https://mlsanity.com/paper/1706.04156/) (2017)
- [Many Paths to Equilibrium: GANs Do Not Need to Decrease a Divergence At Every Step](https://mlsanity.com/paper/1710.08446/) (2017)
- [A case for new neural network smoothness constraints](https://mlsanity.com/paper/2012.07969/) (2020)
- [On gradient regularizers for MMD GANs](https://mlsanity.com/paper/1805.11565/) (2018)
- [Gradient Regularization Improves Accuracy of Discriminative Models](https://mlsanity.com/paper/1712.09936/) (2017)
- [Why Spectral Normalization Stabilizes GANs: Analysis and Improvements](https://mlsanity.com/paper/2009.02773/) (2020)
- [Training Generative Adversarial Networks by Solving Ordinary Differential Equations](https://mlsanity.com/paper/2010.15040/) (2020)
- [Degrees of Freedom in Deep Neural Networks](https://mlsanity.com/paper/1603.09260/) (2016)

## Built on by (most cited first, 7 of 7)

- [Why neural networks find simple solutions: the many regularizers of geometric complexity](https://mlsanity.com/paper/2209.13083/) (2022)
- [Implicit biases in multitask and continual learning from a backward error analysis perspective](https://mlsanity.com/paper/2311.00235/) (2023)
- [The Impact of Geometric Complexity on Neural Collapse in Transfer Learning](https://mlsanity.com/paper/2405.15706/) (2024)
- [Manifold Characteristics That Predict Downstream Task Performance](https://mlsanity.com/paper/2205.07477/) (2022)
- [A Margin-based Multiclass Generalization Bound via Geometric Complexity](https://mlsanity.com/paper/2405.18590/) (2024)
- [Polyra Swarms: A Shape-Based Approach to Machine Learning](https://mlsanity.com/paper/2506.13217/) (2025)
- [Penalizing Localized Dirichlet Energies in Low Rank Tensor Products](https://mlsanity.com/paper/2601.14173/) (2026)

## Abstract (arXiv)

In over-parameterized deep neural networks there can be many possible parameter configurations that fit the training data exactly. However, the properties of these interpolating solutions are poorly understood. We argue that over-parameterized neural networks trained with stochastic gradient descent are subject to a Geometric Occam's Razor; that is, these networks are implicitly regularized by the geometric model complexity. For one-dimensional regression, the geometric model complexity is simply given by the arc length of the function. For higher-dimensional settings, the geometric model complexity depends on the Dirichlet energy of the function. We explore the relationship between this Geometric Occam's Razor, the Dirichlet energy and other known forms of implicit regularization. Finally, for ResNets trained on CIFAR-10, we observe that Dirichlet energy measurements are consistent with the action of this implicit Geometric Occam's Razor.