mlsanityData to 7 Oct 2026

Geometric Regularization from Overparameterization

arXiv 2202.09276 ↗First posted 18 Feb 2022cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

The volume of the distribution of weight sets associated with a loss value may be the source of implicit regularization from overparameterization due to the phenomenon of contracting volume with increasing dimensions for geometric figures demonstrated by hyperspheres. We introduce the geometric regularization conjecture and extract to an explanation for the double descent phenomenon by considering a similar property resulting from shrinking intrinsic dimensionality of the distribution of potential weight set updates available along training path, where if that distribution retracts across a volume verses dimensionality curve peak when approaching the global minima we could expect geometric regularization to re-emerge. We illustrate how data fidelity representational complexity may influence model capacity double descent interpolation thresholds. The existence of epoch and model capacity double descent curves originating from different geometric forms may imply universality of closed n-manifolds having dimensionally adjusted n-sphere volumetric correspondence.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 43 of its 79 references that are in the map. First posted on arXiv 18 Feb 2022.

Builds on

43 of 79 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  8. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  9. 2016Geometric deep learning: going beyond Euclidean datacited by 1,042
  10. 2021Scaling Vision Transformerscited by 993
  11. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  12. 2018On Lazy Training in Differentiable Programmingcited by 671
31 more
  1. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  2. 2020The power of quantum neural networkscited by 377
  3. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  4. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  5. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  6. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  7. 2018To understand deep learning we need to understand kernel learningcited by 186
  8. 2020fastai: A Layered API for Deep Learningcited by 183
  9. 2015The Power of Depth for Feedforward Neural Networkscited by 181
  10. 2021The Principles of Deep Learning Theorycited by 169
  11. 2021A Universal Law of Robustness via Isoperimetrycited by 150
  12. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  13. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  14. 2021Theory of overparametrization in quantum neural networkscited by 100
  15. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  16. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 77
  17. 2017On weight initialization in deep neural networkscited by 67
  18. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  19. 2018ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networkscited by 53
  20. 2019The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descentcited by 46
  21. 2021Heavy Tails in SGD and Compressibility of Overparametrized Neural Networkscited by 44
  22. 2019Exact expressions for double descent and implicit regularization via surrogate random designcited by 35
  23. 2021Learning with Hyperspherical Uniformitycited by 35
  24. 2021Subquadratic Overparameterization for Shallow Neural Networkscited by 26
  25. 2019Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theorycited by 21
  26. 2022Towards Sample-efficient Overparameterized Meta-learningcited by 21
  27. 2021On the Inherent Regularization Effects of Noise Injection During Trainingcited by 18
  28. 2020A scale-dependent notion of effective dimensioncited by 17
  29. 2019Effect of Activation Functions on the Training of Overparametrized Neural Netscited by 14
  30. 2019Intrinsic dimension estimation for locally undersampled datacited by 10
  31. 2021On Data-centric Mythscited by 4

Built on by

Not cited yet.