mlsanityData to 14 Oct 2026

The loss landscape of deep linear neural networks: a second-order analysis

arXiv 2107.13289 ↗First posted 28 Jul 2021math.ST cs.LG stat.THcited by 24

Not on the map. Why?

Abstract (arXiv)

We study the optimization landscape of deep linear neural networks with the square loss. It is known that, under weak assumptions, there are no spurious local minima and no local maxima. However, the existence and diversity of non-strict saddle points, which can play a role in first-order algorithms' dynamics, have only been lightly studied. We go a step further with a full analysis of the optimization landscape at order 2. We characterize, among all critical points, which are global minimizers, strict saddle points, and non-strict saddle points. We enumerate all the associated critical values. The characterization is simple, involves conditions on the ranks of partial matrix products, and sheds some light on global convergence or implicit regularization that have been proved or observed when optimizing linear neural networks. In passing, we provide an explicit parameterization of the set of all global minimizers and exhibit large sets of strict and non-strict saddle points.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 43 of its 56 references that are in the map. First posted on arXiv 28 Jul 2021.

Builds on

43 of 56 references.

References · 43

  1. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  2. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  3. 2019Benign Overfitting in Linear Regressioncited by 549
  4. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  5. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  6. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  7. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  8. 2017How to Escape Saddle Points Efficientlycited by 285
  9. 2014The Loss Surfaces of Multilayer Networkscited by 280
  10. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 243
  11. 2016Deep Learning without Poor Local Minimacited by 234
  12. 2018A mathematical theory of semantic development in deep neural networkscited by 186
31 more
  1. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  2. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
  3. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 119
  4. 2016Identity Matters in Deep Learningcited by 109
  5. 2016A Geometric Analysis of Phase Retrievalcited by 92
  6. 2013Early stopping and non-parametric regression: An optimal data-dependent stopping rulecited by 89
  7. 2017The loss surface of deep and wide neural networkscited by 74
  8. 2019Optimization for deep learning: theory and algorithmscited by 70
  9. 2017Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descentcited by 66
  10. 2018Escaping Saddles with Stochastic Gradientscited by 66
  11. 2017Deep linear neural networks with arbitrary loss: All local minima are globalcited by 65
  12. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  13. 2019Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizerscited by 61
  14. 2020The Global Landscape of Neural Networks: An Overviewcited by 57
  15. 2019Width Provably Matters in Optimization for Deep Linear Neural Networkscited by 52
  16. 2019On Connected Sublevel Sets in Deep Learningcited by 51
  17. 2018Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networkscited by 44
  18. 2019Pure and Spurious Critical Points: a Geometric Study of Linear Networkscited by 33
  19. 2017Global optimality conditions for deep neural networkscited by 31
  20. 2017Depth Creates No Bad Local Minimacited by 29
  21. 2020From Symmetry to Geometry: Tractable Nonconvex Problemscited by 29
  22. 2018Exponential Convergence Time of Gradient Descent for One-Dimensional Deep Linear Neural Networkscited by 22
  23. 2017On the Optimization Landscape of Tensor Decompositionscited by 21
  24. 2018The loss surface of deep linear networks viewed through the algebraic geometry lenscited by 20
  25. 2015Complete Dictionary Recovery over the Spherecited by 19
  26. 2020Training Linear Neural Networks: Non-Local Convergence and Complexity Resultscited by 19
  27. 2019Global Convergence of Gradient Descent for Deep Linear Residual Networkscited by 16
  28. 2018The Global Optimization Geometry of Shallow Linear Neural Networkscited by 11
  29. 2018Learning Deep Models: Critical Points and Local Opennesscited by 10
  30. 2020Optimization Landscape of Tucker Decompositioncited by 10
  31. 2018A Geometric Approach of Gradient Descent Algorithms in Linear Neural Networkscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 3; 2024: 5; 2025: 8; 2026: 6.

24 papers.

Sort:

  1. 2024How do Transformers perform In-Context Autoregressive Learning?cited by 17
  2. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 15
  3. 2024Only Strict Saddles in the Energy Landscape of Predictive Coding Networks?cited by 10
  4. 2022Deep Linear Networks can Benignly Overfit when Shallow Ones Docited by 8
  5. 2023Geometric structure of Deep Learning networks and construction of global mathcal L2 minimizerscited by 6
  6. 2023Geometric structure of shallow neural networks and constructive mathcal L2 cost minimizationcited by 4
  7. 2024Interpretable global minima of deep ReLU neural networks on sequentially separable datacited by 4
  8. 2025Regularization Implies balancedness in the deep linear networkcited by 4
  9. 2024Geometry of Critical Sets and Existence of Saddle Branches for Two-layer Neural Networkscited by 2
  10. 2025A Complete Loss Landscape Analysis of Regularized Deep Matrix Factorizationcited by 2
  11. 2025A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Biascited by 2
  12. 2022A Dynamics Theory of Implicit Regularization in Deep Low-Rank Matrix Factorizationcited by 1
12 more
  1. 2023Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relationcited by 1
  2. 2026Mapping Networkscited by 1
  3. 2024A new Input Convex Neural Network with application to options pricingcited by 0
  4. 2025Impact of Bottleneck Layers and Skip Connections on the Generalization of Linear Denoising Autoencoderscited by 0
  5. 2025Computing frustration and near-monotonicity in deep neural networkscited by 0
  6. 2025AutoBalance: An Automatic Balancing Framework for Training Physics-Informed Neural Networkscited by 0
  7. 2025The Interplay of Statistics and Noisy Optimization: Learning Linear Predictors with Random Data Weightscited by 0
  8. 2026A Geometric Characterization of the Stationary Plateau for Two-Layer Neural Networkscited by 0
  9. 2026Effects of width-dependent model hyperparameters and ℓ2-regularization on the loss landscape of two-layer ReLU networkscited by 0
  10. 2026Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networkscited by 0
  11. 2026From Objectives to What Models Learn: A Landau Theory of Invariant Learningcited by 0
  12. 2026Benign Loss Landscapes Can Coexist with Worst-Case Hardnesscited by 0