Sharpness of Minima in Deep Matrix Factorization
Not on the map. Why?
Abstract (arXiv)
Understanding the geometry of the loss landscape near a minimum is key to explaining the implicit bias of gradient-based methods in non-convex optimization problems such as deep neural network training and deep matrix factorization. A central quantity to characterize this geometry is the maximum eigenvalue of the Hessian of the loss. Currently, its precise role has been obfuscated because no exact expressions for this sharpness measure were known in general settings. In this paper, we present the first exact expression for the maximum eigenvalue of the Hessian of the squared-error loss at any minimizer in deep matrix factorization/deep linear neural network training problems, resolving an open question posed by Mulayoff & Michaeli (2020). This expression reveals a fundamental property of the loss landscape in deep matrix factorization: Having a constant product of the spectral norms of the left and right intermediate factors across layers is a sufficient condition for flatness. Most notably, in both depth-2 matrix and deep overparameterized scalar factorization, we show that this condition is both necessary and sufficient for flatness, which implies that flat minima are spectral-norm balanced even though they are not necessarily Frobenius-norm balanced. To complement our theory, we provide the first empirical characterization of an escape phenomenon during gradient-based training near a minimizer of a deep matrix factorization problem.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 35 of its 44 references that are in the map. First posted on arXiv 30 Sep 2025.
Builds on
35 of 44 references.
References · 35
- 2017Attention Is All You Needcited by 60,361
- 2016Understanding deep learning requires rethinking generalizationcited by 1,674
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
- 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
- 2017Exploring Generalization in Deep Learningcited by 600
- 2019Benign Overfitting in Linear Regressioncited by 549
- 2019Fantastic Generalization Measures and Where to Find Themcited by 513
- 2017Sharp Minima Can Generalize For Deep Netscited by 466
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
- 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
- 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
23 more
- 2017Implicit Regularization in Matrix Factorizationcited by 322
- 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
- 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
- 2014Qualitatively characterizing neural network optimization problemscited by 188
- 2020Directional convergence and alignment in deep learningcited by 178
- 2018Does data interpolation contradict statistical optimality?cited by 95
- 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 75
- 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
- 2017Deep linear neural networks with arbitrary loss: All local minima are globalcited by 65
- 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 63
- 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 56
- 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 50
- 2020Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rankcited by 42
- 2022Flat minima generalize for low-rank matrix recoverycited by 28
- 2018Step Size Matters in Deep Learningcited by 19
- 2024Deep linear networks for regression are implicitly regularized towards flat minimacited by 17
- 2021Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorizationcited by 15
- 2023The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networkscited by 15
- 2024Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizescited by 12
- 2024Characterizing Dynamical Stability of Stochastic Gradient Descent in Overparameterized Learningcited by 8
- 2025Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Regioncited by 7
- 2025Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenoncited by 6
- 2025On the geometry of flat minimacited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.