mlsanityData to 7 Oct 2026

Incremental Learning in Diagonal Linear Networks

arXiv 2208.14673 ↗First posted 31 Aug 2022cs.LG math.OCcited by 27

Not on the map. Why?

Abstract (arXiv)

Diagonal linear networks (DLNs) are a toy simplification of artificial neural networks; they consist in a quadratic reparametrization of linear regression inducing a sparse implicit regularization. In this paper, we describe the trajectory of the gradient flow of DLNs in the limit of small initialization. We show that incremental learning is effectively performed in the limit: coordinates are successively activated, while the iterate is the minimizer of the loss constrained to have support on the active coordinates only. This shows that the sparse implicit regularization of DLNs decreases with time. This work is restricted to the underparametrized regime with anti-correlated features for technical reasons.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 26 of its 43 references that are in the map. First posted on arXiv 31 Aug 2022.

Builds on

26 of 43 references.

References · 26

  1. 2021Convex optimizationcited by 2,479
  2. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  3. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  4. 2017Implicit Regularization in Matrix Factorizationcited by 318
  5. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  6. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  7. 2021Deep learning: a statistical viewpointcited by 239
  8. 2018A mathematical theory of semantic development in deep neural networkscited by 182
  9. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  10. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 122
  11. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  12. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
14 more
  1. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  2. 2019Implicit Regularization for Optimal Sparse Recoverycited by 78
  3. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  4. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  5. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  6. 2021Implicit Regularization in Tensor Factorizationcited by 43
  7. 2020Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rankcited by 42
  8. 2021Implicit Sparse Regularization: The Impact of Depth and Early Stoppingcited by 30
  9. 2022Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisationcited by 29
  10. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  11. 2021More is Less: Inducing Sparsity via Overparameterizationcited by 22
  12. 2021Understanding Deflation Process in Over-parametrized Tensor Decompositioncited by 20
  13. 2021Smooth Bilevel Programming for Sparse Regularizationcited by 14
  14. 2022Implicit Regularization with Polynomial Growth in Deep Tensor Factorizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 5; 2025: 7; 2026: 10.

27 papers.

Sort:

By area:

  1. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 54
  2. 2023Transformers learn through gradual rank increasecited by 52
  3. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 50
  4. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 35
  5. 2024How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networkscited by 29
  6. 2024Synchronization on circles and spheres with nonlinear interactionscited by 17
  7. 2025Learning In-context n-grams with Transformers: Sub-n-grams Are Near-stationary Pointscited by 15
  8. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  9. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  10. 2023Implicit Regularization for Group Sparsitycited by 8
  11. 2026To Use or not to Use Muon: How Simplicity Bias in Optimizers Matterscited by 7
  12. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
15 more
  1. 2026A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuningcited by 5
  2. 2024Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depthcited by 4
  3. 2024When Are Bias-Free ReLU Networks Effectively Linear Networks?cited by 3
  4. 2024Towards Understanding Epoch-wise Double descent in Two-layer Linear Neural Networkscited by 3
  5. 2025Diagonal Linear Networks and the Lasso Regularization Pathcited by 3
  6. 2025Precise Dynamics of Diagonal Linear Networks: A Unifying Analysis by Dynamical Mean-Field Theorycited by 1
  7. 2026Incremental Learning of Sparse Attention Patterns in Transformerscited by 1
  8. 2026Incremental Learning in Mirror Flowscited by 1
  9. 2025Convergence, Sticking and Escape: Stochastic Dynamics Near Critical Points in SGDcited by 0
  10. 2026Implicit Bias in Deep Linear Discriminant Analysiscited by 0
  11. 2026A Theory of Saddle Escape in Deep Nonlinear Networkscited by 0
  12. 2026A lift for input-convex neural net trainingcited by 0
  13. 2026Singular perturbations and hierarchical learning in two-layer neural networkscited by 0
  14. 2026Gradient Flow Dynamics and Implicit Bias of Diagonal Linear Networks under Infinitesimal Initializationcited by 0
  15. 2026Statistical Benefits of Fine-Tuning from Pretrained Initialization in Diagonal Linear Networkscited by 0