mlsanityData to 14 Oct 2026

Deep linear networks for regression are implicitly regularized towards flat minima

arXiv 2405.13456 ↗First posted 22 May 2024stat.ML cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

The largest eigenvalue of the Hessian, or sharpness, of neural networks is a key quantity to understand their optimization dynamics. In this paper, we study the sharpness of deep linear networks for univariate regression. Minimizers can have arbitrarily large sharpness, but not an arbitrarily small one. Indeed, we show a lower bound on the sharpness of minimizers, which grows linearly with depth. We then study the properties of the minimizer found by gradient flow, which is the limit of gradient descent with vanishing learning rate. We show an implicit regularization towards flat minima: the sharpness of the minimizer is no more than a constant times the lower bound. The constant depends on the condition number of the data covariance matrix, but not on width or depth. This result is proven both for a small-scale initialization and a residual initialization. Results of independent interest are shown in both cases. For small-scale initialization, we show that the learned weight matrices are approximately rank-one and that their singular vectors align. For residual initialization, convergence of the gradient flow for a Gaussian initialization of the residual network is proven. Numerical experiments illustrate our results and connect them to gradient descent with non-vanishing learning rate.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 46 of its 57 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

46 of 57 references.

References · 46

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  5. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  6. 2018On Lazy Training in Differentiable Programmingcited by 675
  7. 2017Exploring Generalization in Deep Learningcited by 600
  8. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  9. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  10. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  11. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  12. 2017Implicit Regularization in Matrix Factorizationcited by 322
34 more
  1. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  2. 2017Three Factors Influencing Minima in SGDcited by 273
  3. 2017High-dimensional dynamics of generalization error in neural networkscited by 241
  4. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  5. 2018A mathematical theory of semantic development in deep neural networkscited by 186
  6. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  7. 2020Directional convergence and alignment in deep learningcited by 178
  8. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  9. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
  10. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 135
  11. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
  12. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  13. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  14. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  15. 2023Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networkscited by 97
  16. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 92
  17. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 85
  18. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 85
  19. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  20. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 75
  21. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 63
  22. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  23. 2018Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networkscited by 44
  24. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  25. 2019How to Initialize your Network? Robust Initialization for WeightNorm & ResNetscited by 36
  26. 2024Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiencycited by 35
  27. 2020On the Global Convergence of Training Deep Linear ResNetscited by 29
  28. 2022Do Residual Neural Networks discretize Neural Ordinary Differential Equations?cited by 27
  29. 2023Implicit regularization of deep residual networks towards neural ODEscited by 20
  30. 2022Scaling ResNets in the Large-depth Regimecited by 18
  31. 2023The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networkscited by 11
  32. 2019Stabilize Deep ResNet with A Sharp Scaling Factor τcited by 10
  33. 2023On progressive sharpening, flat minima and generalisationcited by 7
  34. 2023Fast convergence to non-isolated minima: four equivalent conditions for C2 functionscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 7; 2026: 7.

17 papers.

Sort:

  1. 2024The Optimization Landscape of SGD Across the Feature Learning Strengthcited by 13
  2. 2025Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?cited by 12
  3. 2025Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Regioncited by 7
  4. 2025Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and Morecited by 5
  5. 2024When Are Bias-Free ReLU Networks Effectively Linear Networks?cited by 4
  6. 2026Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learningcited by 3
  7. 2024Optimization Insights into Deep Diagonal Linear Networkscited by 2
  8. 2025The Features at Convergence Theorem: a first-principles alternative to the Neural Feature Ansatz for how networks learn representationscited by 2
  9. 2025Sharpness of Minima in Deep Matrix Factorizationcited by 2
  10. 2026The Impact of Anisotropic Covariance Structure on the Training Dynamics and Generalization Error of Linear Networkscited by 2
  11. 2026Does SGD Seek Flatness or Sharpness? An Exactly Solvable Modelcited by 1
  12. 2026Large Spikes in Stochastic Gradient Descent: A Large-Deviations Viewcited by 1
5 more
  1. 2026Wolkowicz-Styan Upper Bound on the Hessian Eigenspectrum for Cross-Entropy Loss in Nonlinear Smooth Neural Networkscited by 1
  2. 2026Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minimacited by 1
  3. 2025Convergence of optimizers implies eigenvalues filtering at equilibriumcited by 0
  4. 2025An Equivariance Toolbox for Learning Dynamicscited by 0
  5. 2026Gradient Descent with Large Step Size Restores Symmetry in Deep Linear Networks with Multi-Pathwaycited by 0