mlsanityData to 14 Oct 2026

Diagonal Linear Networks and the Lasso Regularization Path

arXiv 2509.18766 ↗First posted 23 Sep 2025cs.LG math.OC stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

Diagonal linear networks are neural networks with linear activation and diagonal weight matrices. Their theoretical interest is that their implicit regularization can be rigorously analyzed: from a small initialization, the training of diagonal linear networks converges to the linear predictor with minimal 1-norm among minimizers of the training loss. In this paper, we deepen this analysis showing that the full training trajectory of diagonal linear networks is closely related to the lasso regularization path. In this connection, the training time plays the role of an inverse regularization parameter. Both rigorous results and simulations are provided to illustrate this conclusion. Under a monotonicity assumption on the lasso regularization path, the connection is exact while in the general case, we show an approximate connection.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 20 of its 35 references that are in the map. First posted on arXiv 23 Sep 2025.

Builds on

20 of 35 references.

References · 20

  1. 2021Convex optimizationcited by 2,515
  2. 2014Convex Optimization: Algorithms and Complexitycited by 575
  3. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  4. 2017Implicit Regularization in Matrix Factorizationcited by 322
  5. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  6. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
  7. 2021Deep learning: a statistical viewpointcited by 240
  8. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
  9. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
  10. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  11. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
  12. 2019Implicit Regularization for Optimal Sparse Recoverycited by 80
8 more
  1. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 70
  2. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 57
  3. 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 39
  4. 2021Implicit Sparse Regularization: The Impact of Depth and Early Stoppingcited by 30
  5. 2022Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisationcited by 29
  6. 2022Incremental Learning in Diagonal Linear Networkscited by 28
  7. 2019High-Dimensional Linear Regression via Implicit Regularizationcited by 25
  8. 2021More is Less: Inducing Sparsity via Overparameterizationcited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 15
  2. 2026Incremental Learning in Mirror Flowscited by 2
  3. 2026Gradient Flow Dynamics and Implicit Bias of Diagonal Linear Networks under Infinitesimal Initializationcited by 1
  4. 2026Grokking through the Lens of Minimum-Norm Interpolationcited by 0