mlsanityData to 7 Oct 2026

Exact Solutions of a Deep Linear Network

arXiv 2202.04777 ↗First posted 10 Feb 2022stat.ML cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

This work finds the analytical expression of the global minima of a deep linear network with weight decay and stochastic neurons, a fundamental model for understanding the landscape of neural networks. Our result implies that the origin is a special point in deep neural network loss landscape where highly nonlinear phenomenon emerges. We show that weight decay strongly interacts with the model architecture and can create bad minima at zero in a network with more than 1 hidden layer, qualitatively different from a network with only 1 hidden layer. Practically, our result implies that common deep learning initialization methods are insufficient to ease the optimization of neural networks in general.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 27 of its 56 references that are in the map. First posted on arXiv 10 Feb 2022.

Builds on

27 of 56 references.

References · 27

  1. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  2. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  3. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  4. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  5. 2014The Loss Surfaces of Multilayer Networkscited by 278
  6. 2016Deep Learning without Poor Local Minimacited by 233
  7. 2019Diagnosing and Enhancing VAE Modelscited by 192
  8. 2019Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapsecited by 120
  9. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  10. 2018A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillationcited by 110
  11. 2016Identity Matters in Deep Learningcited by 107
  12. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
15 more
  1. 2017Deep linear neural networks with arbitrary loss: All local minima are globalcited by 64
  2. 2020Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descentcited by 36
  3. 2017Depth Creates No Bad Local Minimacited by 29
  4. 2021Power-law escape rate of SGDcited by 29
  5. 2018Small nonlinearities in activation functions create bad local minima in neural networkscited by 28
  6. 2020Dropout: Explicit Forms and Capacity Controlcited by 27
  7. 2018The loss surface of deep linear networks viewed through the algebraic geometry lenscited by 20
  8. 2020Piecewise linear activations substantially shape the loss surfaces of neural networkscited by 19
  9. 2017Dropout as a Low-Rank Regularizer for Matrix Factorizationcited by 14
  10. 2019On Dropout and Nuclear Norm Regularizationcited by 12
  11. 2021Variational autoencoders in the presence of low-dimensional data: landscape and implicit biascited by 11
  12. 2021SGD with a Constant Large Learning Rate Can Converge to Local Maximacited by 8
  13. 2021Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activationscited by 7
  14. 2017How regularization affects the critical points in linear networkscited by 5
  15. 2022Stochastic Neural Networks with Infinite Width are Deterministiccited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 3; 2024: 10; 2025: 3; 2026: 5.

24 papers.

Sort:

  1. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 46
  2. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  3. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 38
  4. 2024Weight decay induces low-rank attention layerscited by 27
  5. 2023Symmetry Induces Structure and Constraint of Learningcited by 25
  6. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  7. 2022What shapes the loss landscape of self-supervised learning?cited by 19
  8. 2024Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descentcited by 19
  9. 2022Posterior Collapse of a Linear Latent Variable Modelcited by 18
  10. 2024Plastic Learning with Deep Fourier Featurescited by 14
  11. 2024Only Strict Saddles in the Energy Landscape of Predictive Coding Networks?cited by 10
  12. 2025Learning on a Razor's Edge: Identifiability and Singularity of Polynomial Neural Networkscited by 9
12 more
  1. 2025Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learningcited by 8
  2. 2026To Use or not to Use Muon: How Simplicity Bias in Optimizers Matterscited by 7
  3. 2022Exact Phase Transitions in Deep Learningcited by 5
  4. 2024Calabi-Yau metrics through Grassmannian learning and Donaldson's algorithmcited by 4
  5. 2023Representation Learning Dynamics of Self-Supervised Modelscited by 3
  6. 2024Three Mechanisms of Feature Learning in a Linear Networkcited by 3
  7. 2024When Are Bias-Free ReLU Networks Effectively Linear Networks?cited by 3
  8. 2025The Features at Convergence Theorem: a first-principles alternative to the Neural Feature Ansatz for how networks learn representationscited by 2
  9. 2024Neural Networks with (Low-Precision) Polynomial Approximations: New Insights and Techniques for Accuracy Improvementcited by 1
  10. 2026Does SGD Seek Flatness or Sharpness? An Exactly Solvable Modelcited by 1
  11. 2026Singular Learning and Occam's Razor in Deep Monomial Networkscited by 0
  12. 2026Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networkscited by 0