mlsanityData to 7 Oct 2026

Implicit Bias of MSE Gradient Optimization in Underparameterized Neural Networks

arXiv 2201.04738 ↗First posted 12 Jan 2022stat.ML cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

We study the dynamics of a neural network in function space when optimizing the mean squared error via gradient flow. We show that in the underparameterized regime the network learns eigenfunctions of an integral operator TK^∞ determined by the Neural Tangent Kernel (NTK) at rates corresponding to their eigenvalues. For example, for uniformly distributed data on the sphere Sd - 1 and rotation invariant weight distributions, the eigenfunctions of TK^∞ are the spherical harmonics. Our results can be understood as describing a spectral bias in the underparameterized regime. The proofs use the concept of "Damped Deviations", where deviations of the NTK matter less for eigendirections with large eigenvalues due to the occurence of a damping factor. Aside from the underparameterized regime, the damped deviations point-of-view can be used to track the dynamics of the empirical risk in the overparameterized setting, allowing us to extend certain results in the literature. We conclude that damped deviations offers a simple and unifying perspective of the dynamics when optimizing the squared error.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 32 of its 49 references that are in the map. First posted on arXiv 12 Jan 2022.

Builds on

32 of 49 references.

References · 32

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2018On the Spectral Bias of Neural Networkscited by 1,321
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  5. 2018On Lazy Training in Differentiable Programmingcited by 671
  6. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  7. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  8. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  9. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  10. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  11. 2017Implicit Regularization in Matrix Factorizationcited by 318
  12. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
20 more
  1. 2018Training behavior of deep neural network in frequency domaincited by 174
  2. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 173
  3. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  4. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  5. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
  6. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  7. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  8. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  9. 2017Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapescited by 84
  10. 2018On the Convergence Rate of Training Recurrent Neural Networkscited by 82
  11. 2019Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobiancited by 67
  12. 2020Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topologycited by 66
  13. 2019Quadratic Suffices for Over-parametrization via Matrix Chernoff Boundcited by 58
  14. 2019A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamicscited by 49
  15. 2019Gradient Dynamics of Shallow Univariate ReLU Networkscited by 46
  16. 2017Geometry of Optimization and Implicit Regularization in Deep Learningcited by 39
  17. 2019A type of generalization error induced by initialization in deep neural networkscited by 33
  18. 2019On Learning Over-parameterized Neural Networks: A Functional Approximation Perspectivecited by 28
  19. 2020Optimization Theory for ReLU Neural Networks Trained with Normalization Layerscited by 15
  20. 2017Not-So-Random Featurescited by 5

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 1; 2024: 3; 2025: 2.

9 papers.

  1. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  2. 2024A distributional simplicity bias in the learning dynamics of transformerscited by 23
  3. 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
  4. 2022Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regimecited by 11
  5. 2024Simplicity Bias of Two-Layer Networks beyond Linearly Separable Datacited by 11
  6. 2023On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspectivecited by 6
  7. 2024Optimal Rates of Convergence for Entropy Regularization in Discounted Markov Decision Processescited by 1
  8. 2025Make Optimization Once and for All with Fine-grained Guidancecited by 1
  9. 2025Progressive Translation of H&E to IHC with Enhanced Structural Fidelitycited by 0