Gradient Flow Dynamics and Implicit Bias of Diagonal Linear Networks under Infinitesimal Initialization
Not on the map. Why?
Abstract (arXiv)
We study the gradient flow dynamics of diagonal linear networks for regression tasks under infinitesimal initialization. Extending Theorem 1 from Pesme & Flammarion (2023), we generalize the analysis to both deep diagonal linear networks and a broader class of two-layer diagonal linear networks (as defined in Definition 4.1). Specifically, we demonstrate that the training trajectories of these models can be equivalently characterized by the proposed Algorithm 1. We further prove that this algorithm converges to the solution of a modified l1 norm minimization problem. As a result, we establish that the implicit bias of both network architectures corresponds to a modified l1 norm in the regime of infinitesimal initialization. Additionally, we provide insights into the underlying mechanisms governing these dynamics by identifying the Structural Invariant Manifold (SIM) (Zhao et al., 2026) as the key geometric structure that shapes the learning process.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 25 of its 33 references that are in the map. First posted on arXiv 14 Jul 2026.
Builds on
25 of 33 references.
References · 25
- 2016Understanding deep learning requires rethinking generalizationcited by 1,674
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
- 2017Exploring Generalization in Deep Learningcited by 600
- 2017Implicit Regularization in Matrix Factorizationcited by 322
- 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
- 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
- 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
- 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 127
- 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
- 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
- 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 85
- 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 76
13 more
- 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 70
- 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 57
- 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 45
- 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 39
- 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 36
- 2022Incremental Learning in Diagonal Linear Networkscited by 28
- 2023Symmetry Induces Structure and Constraint of Learningcited by 25
- 2024Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descentcited by 19
- 2024Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networkscited by 9
- 2025Saddle-To-Saddle Dynamics in Deep ReLU Networks: Low-Rank Bias in the First Saddle Escapecited by 4
- 2025Diagonal Linear Networks and the Lasso Regularization Pathcited by 4
- 2024Connectivity Shapes Implicit Regularization in Matrix Factorization Models for Matrix Completioncited by 3
- 2024Optimization Insights into Deep Diagonal Linear Networkscited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.