There is a Singularity in the Loss Landscape
Not on the map. Why?
Abstract (arXiv)
Despite the widespread adoption of neural networks, their training dynamics remain poorly understood. We show experimentally that as the size of the dataset increases, a point forms where the magnitude of the gradient of the loss becomes unbounded. Gradient descent rapidly brings the network close to this singularity in parameter space, and further training takes place near it. This singularity explains a variety of phenomena recently observed in the Hessian of neural network loss functions, such as training on the edge of stability and the concentration of the gradient in a top subspace. Once the network approaches the singularity, the top subspace contributes little to learning, even though it constitutes the majority of the gradient.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 20 of its 26 references that are in the map. First posted on arXiv 12 Jan 2022.
Builds on
20 of 26 references.
References · 20
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2016Understanding deep learning requires rethinking generalizationcited by 1,657
- 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
- 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
- 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
- 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
8 more
- 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
- 2018Gradient Descent Happens in a Tiny Subspacecited by 208
- 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
- 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
- 2020The Recurrent Neural Tangent Kernelcited by 67
- 2019Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspectivecited by 26
- 2021A Neural Tangent Kernel Perspective of GANscited by 21
- 2020Towards an Understanding of Residual Networks Using Neural Tangent Hierarchy (NTH)cited by 4
Built on by
Citing papers per year: 2025: 1.
1 paper.