mlsanityData to 14 Oct 2026

Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimes

arXiv 2405.17580 ↗First posted 27 May 2024cs.LG cs.AI stat.MLcited by 18

Not on the map. Why?

Abstract (arXiv)

The training dynamics of linear networks are well studied in two distinct setups: the lazy regime and balanced/active regime, depending on the initialization and width of the network. We provide a surprisingly simple unifying formula for the evolution of the learned matrix that contains as special cases both lazy and balanced regimes but also a mixed regime in between the two. In the mixed regime, a part of the network is lazy while the other is balanced. More precisely the network is lazy along singular values that are below a certain threshold and balanced along those that are above the same threshold. At initialization, all singular values are lazy, allowing for the network to align itself with the task, so that later in time, when some of the singular value cross the threshold and become active they will converge rapidly (convergence in the balanced regime is notoriously difficult in the absence of alignment). The mixed regime is the `best of both worlds': it converges from any random initialization (in contrast to balanced dynamics which require special initialization), and has a low rank bias (absent in the lazy dynamics). This allows us to prove an almost complete phase diagram of training behavior as a function of the variance at initialization and the width, for a MSE training task.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 38 of its 54 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

38 of 54 references.

References · 38

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  4. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  5. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  6. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  7. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  8. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  9. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  10. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  11. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  12. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
26 more
  1. 2017High-dimensional dynamics of generalization error in neural networkscited by 241
  2. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  3. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  4. 2018A mathematical theory of semantic development in deep neural networkscited by 186
  5. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  6. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  7. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
  8. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 143
  9. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 142
  10. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
  11. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 114
  12. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
  13. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
  14. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 85
  15. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 70
  16. 2021The staircase property: How hierarchical structure can guide deep learningcited by 62
  17. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 57
  18. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 42
  19. 2020On the training dynamics of deep networks with L2 regularizationcited by 39
  20. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  21. 2020On the Global Convergence of Training Deep Linear ResNetscited by 29
  22. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 29
  23. 2022Infinite-width limit of deep linear neural networkscited by 20
  24. 2023How Over-Parameterization Slows Down Gradient Descent in Matrix Sensing: The Curses of Symmetry and Initializationcited by 18
  25. 2022Feature Learning in L2-regularized DNNs: Attraction/Repulsion and Sparsitycited by 17
  26. 2023Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoffcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 6; 2026: 4.

18 papers.

Sort:

  1. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 56
  2. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 49
  3. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 42
  4. 2024ResearchTown: Simulator of Human Research Communitycited by 37
  5. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 16
  6. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 15
  7. 2024The Optimization Landscape of SGD Across the Feature Learning Strengthcited by 13
  8. 2025Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)cited by 9
  9. 2025On the Role of Label Noise in the Feature Learning Processcited by 4
  10. 2025Saddle-To-Saddle Dynamics in Deep ReLU Networks: Low-Rank Bias in the First Saddle Escapecited by 4
  11. 2026Towards Worst-Case Guarantees with Scale-Aware Interpretabilitycited by 3
  12. 2024Geometric Inductive Biases of Deep Networks: The Role of Data and Architecturecited by 2
6 more
  1. 2025Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamicscited by 2
  2. 2026A unified theory of feature learning in RNNs and DNNscited by 2
  3. 2025Feature learning is decoupled from generalization in high capacity neural networkscited by 1
  4. 2024A Generalization Bound for Nearly-Linear Networkscited by 0
  5. 2026The Implicit Bias of Depth: From Neural Collapse to Softmax Codescited by 0
  6. 2026A prism hierarchy of learning regimes in large linear autoencoderscited by 0