mlsanityData to 7 Oct 2026

Saddle-to-Saddle Dynamics in Diagonal Linear Networks

arXiv 2304.00488 ↗First posted 2 Apr 2023cs.LG math.OCcited by 54

Not on the map. Why?

Abstract (arXiv)

In this paper we fully describe the trajectory of gradient flow over diagonal linear networks in the limit of vanishing initialisation. We show that the limiting flow successively jumps from a saddle of the training loss to another until reaching the minimum ℓ1-norm solution. This saddle-to-saddle dynamics translates to an incremental learning process as each saddle corresponds to the minimiser of the loss constrained to an active set outside of which the coordinates must be zero. We explicitly characterise the visited saddles as well as the jumping times through a recursive algorithm reminiscent of the LARS algorithm used for computing the Lasso path. Our proof leverages a convenient arc-length time-reparametrisation which enables to keep track of the heteroclinic transitions between the jumps. Our analysis requires negligible assumptions on the data, applies to both under and overparametrised settings and covers complex cases where there is no monotonicity of the number of active coordinates. We provide numerical experiments to support our findings.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 31 of its 67 references that are in the map. First posted on arXiv 2 Apr 2023.

Builds on

31 of 67 references.

References · 31

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2018On Lazy Training in Differentiable Programmingcited by 671
  4. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  5. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  6. 2017Implicit Regularization in Matrix Factorizationcited by 318
  7. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  8. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  9. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  10. 2016Deep Learning without Poor Local Minimacited by 233
  11. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  12. 2018A mathematical theory of semantic development in deep neural networkscited by 182
19 more
  1. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  2. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  3. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 122
  4. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  5. 2017Implicit Regularization in Deep Learningcited by 90
  6. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  7. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  8. 2016Global Optimality of Local Search for Low Rank Matrix Recoverycited by 81
  9. 2019Implicit Regularization for Optimal Sparse Recoverycited by 78
  10. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  11. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  12. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  13. 2021Implicit Regularization in Tensor Factorizationcited by 43
  14. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 40
  15. 2016Non-square matrix sensing without spurious local minima via the Burer-Monteiro approachcited by 38
  16. 2022Algorithmic Regularization in Model-free Overparametrized Asymmetric Matrix Factorizationcited by 34
  17. 2019Exponentiated Gradient Meets Gradient Descentcited by 32
  18. 2022Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisationcited by 29
  19. 2022Incremental Learning in Diagonal Linear Networkscited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 15; 2025: 13; 2026: 16.

54 papers.

Sort:

By area:

  1. 2023On Learning Gaussian Multi-index Models with Gradient Flowcited by 53
  2. 2023Transformers learn through gradual rank increasecited by 52
  3. 2023SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problemcited by 36
  4. 2024How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networkscited by 29
  5. 2025The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Trainingcited by 27
  6. 2025The emergence of sparse attention: impact of data distribution and benefits of repetitioncited by 25
  7. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  8. 2023Understanding Unimodal Bias in Multimodal Deep Linear Networkscited by 18
  9. 2024Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimescited by 18
  10. 2025Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regimecited by 18
  11. 2024Synchronization on circles and spheres with nonlinear interactionscited by 17
  12. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
38 more
  1. 2024Implicit Regularization of Gradient Flow on One-Layer Softmax Attentioncited by 16
  2. 2023Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networkscited by 15
  3. 2025Learning In-context n-grams with Transformers: Sub-n-grams Are Near-stationary Pointscited by 15
  4. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  5. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  6. 2025Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorizationcited by 13
  7. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  8. 2023Robust Implicit Regularization via Weight Normalizationcited by 10
  9. 2024When Representations Align: Universality in Representation Learning Dynamicscited by 10
  10. 2024Directional Convergence Near Small Initializations and Saddles in Two-Homogeneous Neural Networkscited by 10
  11. 2025Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)cited by 8
  12. 2025A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Modelscited by 8
  13. 2024Implicit Bias of Mirror Flow on Separable Datacited by 7
  14. 2026Sequential Group Composition: A Window into the Mechanics of Deep Learningcited by 7
  15. 2024Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embeddingcited by 6
  16. 2024Swing-by Dynamics in Concept Learning and Compositional Generalizationcited by 5
  17. 2025Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Lawscited by 5
  18. 2026A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuningcited by 5
  19. 2023Implicit regularization in AI meets generalized hardness of approximation in optimization -- Sharp results for diagonal linear networkscited by 4
  20. 2023Achieving Margin Maximization Exponentially Fast via Progressive Norm Rescalingcited by 4
  21. 2024Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depthcited by 4
  22. 2025Saddle-To-Saddle Dynamics in Deep ReLU Networks: Low-Rank Bias in the First Saddle Escapecited by 4
  23. 2024When Are Bias-Free ReLU Networks Effectively Linear Networks?cited by 3
  24. 2024Towards Understanding Epoch-wise Double descent in Two-layer Linear Neural Networkscited by 3
  25. 2025Diagonal Linear Networks and the Lasso Regularization Pathcited by 3
  26. 2023Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapultscited by 2
  27. 2026Never Saddle for Reparameterized Steepest Descent as Mirror Flowcited by 2
  28. 2024Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Featurescited by 1
  29. 2024A Mechanistic Study of Transformers Training Dynamicscited by 1
  30. 2025Precise Dynamics of Diagonal Linear Networks: A Unifying Analysis by Dynamical Mean-Field Theorycited by 1
  31. 2026Incremental Learning of Sparse Attention Patterns in Transformerscited by 1
  32. 2026Specialization of softmax attention heads: insights from the high-dimensional single-location modelcited by 1
  33. 2026Incremental Learning in Mirror Flowscited by 1
  34. 2026On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGDcited by 0
  35. 2026Hard labels sampled from sparse targets mislead rotation invariant algorithmscited by 0
  36. 2026A Theory of Saddle Escape in Deep Nonlinear Networkscited by 0
  37. 2026Singular perturbations and hierarchical learning in two-layer neural networkscited by 0
  38. 2026Gradient Flow Dynamics and Implicit Bias of Diagonal Linear Networks under Infinitesimal Initializationcited by 0

50 most cited of 54 All 54 (JSON)