mlsanityData to 14 Oct 2026

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

arXiv 1312.6120 ↗First posted 20 Dec 2013cs.NE cond-mat.dis-nn cs.CVcited by 763

Not on the map. Why?

Abstract (arXiv)

Despite the widespread practical success of deep learning methods, our theoretical understanding of the dynamics of learning in deep neural networks remains quite sparse. We attempt to bridge the gap between the theory and practice of deep learning by systematically analyzing learning dynamics for the restricted case of deep linear neural networks. Despite the linearity of their input-output map, such networks have nonlinear gradient descent dynamics on weights that change with the addition of each new hidden layer. We show that deep linear networks exhibit nonlinear learning phenomena similar to those seen in simulations of nonlinear networks, including long plateaus followed by rapid transitions to lower error solutions, and faster convergence from greedy unsupervised pretraining initial conditions than from random initial conditions. We provide an analytical description of these phenomena by finding new exact solutions to the nonlinear dynamics of deep learning. Our theoretical analysis also reveals the surprising finding that as the depth of a network approaches infinity, learning speed can nevertheless remain finite: for a special class of initial conditions on the weights, very deep networks incur only a finite, depth independent, delay in learning speed relative to shallow networks. We show that, under certain conditions on the training data, unsupervised pretraining can find this special class of initial conditions, while scaled random Gaussian initializations cannot. We further exhibit a new class of random orthogonal initial conditions on weights that, like unsupervised pre-training, enjoys depth independent learning times. We further show that these initial conditions also lead to faithful propagation of gradients even in deep nonlinear networks, as long as they operate in a special regime known as the edge of chaos.

Key facts. Cited by 763 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Dec 2013.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 2; 2021: 105; 2022: 119; 2023: 129; 2024: 106; 2025: 127; 2026: 175.

763 papers.

Sort:

By area:

  1. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  2. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  3. 2022Transformer for Partial Differential Equations' Operator Learningcited by 283
  4. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  5. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 202
  6. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  7. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 182
  8. 2023Birth of a Transformer: A Memory Viewpointcited by 161
  9. 2021Towards Understanding Knowledge Distillationcited by 158
  10. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
  11. 2024Barren Plateaus in Variational Quantum Computingcited by 133
  12. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 122
38 more
  1. 2022Hyperbolic Vision Transformers: Combining Improvements in Metric Learningcited by 111
  2. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 108
  3. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 107
  4. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  5. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
  6. 2023The Expressive Power of Low-Rank Adaptationcited by 102
  7. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  8. 2022Brain Network Transformercited by 94
  9. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 90
  10. 2022Learning to Learn with Generative Models of Neural Network Checkpointscited by 86
  11. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 85
  12. 2021Proof-of-Learning: Definitions and Practicecited by 84
  13. 2021On the geometry of generalization and memorization in deep neural networkscited by 73
  14. 2021Causal Influence Detection for Improving Efficiency in Reinforcement Learningcited by 71
  15. 2023How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelcited by 71
  16. 2022Overview frequency principle/spectral bias in deep learningcited by 69
  17. 2021The staircase property: How hierarchical structure can guide deep learningcited by 62
  18. 2021Initialization and Regularization of Factorized Neural Layerscited by 58
  19. 2024Normalization and effective learning rates in reinforcement learningcited by 58
  20. 2023Learning time-scales in two-layers neural networkscited by 57
  21. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 56
  22. 2021Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depthcited by 55
  23. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 54
  24. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
  25. 2021A Survey on Green Deep Learningcited by 52
  26. 2023Transformers learn through gradual rank increasecited by 52
  27. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 51
  28. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  29. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  30. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 49
  31. 2022Deep Constrained Least Squares for Blind Image Super-Resolutioncited by 47
  32. 2024Mixture-of-Subspaces in Low-Rank Adaptationcited by 47
  33. 2021Fast Certified Robust Training with Short Warmupcited by 46
  34. 2022On Evaluation Metrics for Graph Generative Modelscited by 46
  35. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 46
  36. 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 45
  37. 2021GradInit: Learning to Initialize Neural Networks for Stable and Efficient Trainingcited by 44
  38. 2023Hypernetwork-based Meta-Learning for Low-Rank Physics-Informed Neural Networkscited by 44

50 most cited of 763 All 763 (JSON)