mlsanityData to 7 Oct 2026

Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rank

arXiv 2305.16038 ↗First posted 25 May 2023cs.LG cs.AI stat.MLcited by 28

Not on the map. Why?

Abstract (arXiv)

The L2-regularized loss of Deep Linear Networks (DLNs) with more than one hidden layers has multiple local minima, corresponding to matrices with different ranks. In tasks such as matrix completion, the goal is to converge to the local minimum with the smallest rank that still fits the training data. While rank-underestimating minima can be avoided since they do not fit the data, GD might get stuck at rank-overestimating minima. We show that with SGD, there is always a probability to jump from a higher rank minimum to a lower rank one, but the probability of jumping back is zero. More precisely, we define a sequence of sets B1subset B2subsetcdotssubset BR so that Br contains all minima of rank r or less (and not more) that are absorbing for small enough ridge parameters λ and learning rates η: SGD has prob. 0 of leaving Br, and from any starting point there is a non-zero prob. for SGD to go in Br.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 24 of its 35 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

24 of 35 references.

References · 24

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  4. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  5. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  6. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  7. 2017Three Factors Influencing Minima in SGDcited by 270
  8. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  9. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  10. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  11. 2020Directional convergence and alignment in deep learningcited by 172
  12. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
12 more
  1. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
  2. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  3. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  4. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 113
  5. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 108
  6. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  7. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  8. 2014Guaranteed Matrix Completion via Non-convex Factorizationcited by 81
  9. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 70
  10. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  11. 2022Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisationcited by 29
  12. 2022Feature Learning in L2-regularized DNNs: Attraction/Repulsion and Sparsitycited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 12; 2025: 7; 2026: 7.

28 papers.

Sort:

  1. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 38
  2. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
  3. 2024Weight decay induces low-rank attention layerscited by 27
  4. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  5. 2025LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)cited by 19
  6. 2024Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimescited by 18
  7. 2023Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoffcited by 17
  8. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 16
  9. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 14
  10. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 12
  11. 2025AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMscited by 12
  12. 2024Provable Emergence of Deep Neural Collapse and Low-Rank Bias in L2-Regularized Nonlinear Networkscited by 10
16 more
  1. 2025Learning on a Razor's Edge: Identifiability and Singularity of Polynomial Neural Networkscited by 9
  2. 2025Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)cited by 8
  3. 2024DeNetDM: Debiasing by Network Depth Modulationcited by 6
  4. 2024How DNNs break the Curse of Dimensionality: Compositionality and Symmetry Learningcited by 6
  5. 2024Can Implicit Bias Imply Adversarial Robustness?cited by 5
  6. 2026Noise-Driven Escape from Metastable Phases explains Grokking in Deep Neural Networkscited by 5
  7. 2025Saddle-To-Saddle Dynamics in Deep ReLU Networks: Low-Rank Bias in the First Saddle Escapecited by 4
  8. 2026Beyond Neural Collapse: Task-Intrinsic Geometry Governs Neural Representations in Modular Arithmeticcited by 2
  9. 2024Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNetscited by 1
  10. 2024When narrower is better: the narrow width limit of Bayesian parallel branching neural networkscited by 1
  11. 2025A Complete Loss Landscape Analysis of Regularized Deep Matrix Factorizationcited by 1
  12. 2025Conv4Rec: A 1-by-1 Convolutional AutoEncoder for User Profiling through Joint Analysis of Implicit and Explicit Feedbackscited by 1
  13. 2026On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGDcited by 0
  14. 2026Stochastic Gradient Descent in the Saddle-to-Saddle Regime of Deep Linear Networkscited by 0
  15. 2026Singular Learning and Occam's Razor in Deep Monomial Networkscited by 0
  16. 2026Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networkscited by 0