mlsanityData to 7 Oct 2026

Implicit Sparse Regularization: The Impact of Depth and Early Stopping

arXiv 2108.05574 ↗First posted 12 Aug 2021stat.ML cs.LGcited by 30

Not on the map. Why?

Abstract (arXiv)

In this paper, we study the implicit bias of gradient descent for sparse regression. We extend results on regression with quadratic parametrization, which amounts to depth-2 diagonal linear networks, to more general depth-N networks, under more realistic settings of noise and correlated designs. We show that early stopping is crucial for gradient descent to converge to a sparse model, a phenomenon that we call implicit sparse regularization. This result is in sharp contrast to known results for noiseless and uncorrelated-design cases. We characterize the impact of depth and early stopping and show that for a general depth parameter N, gradient descent with early stopping achieves minimax optimal sparse recovery with sufficiently small initialization and step size. In particular, we show that increasing depth enlarges the scale of working initialization and the early-stopping window so that this implicit sparse regularization effect is more likely to take place.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 21 of its 34 references that are in the map. First posted on arXiv 12 Aug 2021.

Builds on

21 of 34 references.

References · 21

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  4. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  5. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  6. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  7. 2017Implicit Regularization in Matrix Factorizationcited by 318
  8. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  9. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  10. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  11. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 158
  12. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
9 more
  1. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 122
  2. 2019Harmless interpolation of noisy data in regressioncited by 121
  3. 2013Early stopping and non-parametric regression: An optimal data-dependent stopping rulecited by 86
  4. 2019Implicit Regularization for Optimal Sparse Recoverycited by 78
  5. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 70
  6. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  7. 2019Gradient Dynamics of Shallow Univariate ReLU Networkscited by 46
  8. 2017Geometry of Optimization and Implicit Regularization in Deep Learningcited by 39
  9. 2018Iterate averaging as regularization for stochastic gradient descentcited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 7; 2024: 7; 2025: 6; 2026: 5.

30 papers.

Sort:

  1. 2022Robust Training under Label Noise by Over-parameterizationcited by 93
  2. 2022Incremental Learning in Diagonal Linear Networkscited by 27
  3. 2021More is Less: Inducing Sparsity via Overparameterizationcited by 22
  4. 2024Implicit Regularization of Gradient Flow on One-Layer Softmax Attentioncited by 16
  5. 2025Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetriescited by 13
  6. 2024Mask in the Mirror: Implicit Sparsificationcited by 12
  7. 2023Robust Implicit Regularization via Weight Normalizationcited by 10
  8. 2023Implicit Regularization for Group Sparsitycited by 8
  9. 2025Sign-In to the Lottery: Reparameterizing Sparse Training From Scratchcited by 6
  10. 2023Balance is Essence: Accelerating Sparse Training via Adaptive Gradient Correctioncited by 5
  11. 2023Implicit Regularization Leads to Benign Overfitting for Sparse Linear Regressioncited by 4
  12. 2023Implicit regularization in AI meets generalized hardness of approximation in optimization -- Sharp results for diagonal linear networkscited by 4
18 more
  1. 2024FTMixer: Frequency and Time Domain Representations Fusion for Time Series Modelingcited by 4
  2. 2022Towards Data-Algorithm Dependent Generalization: a Case Study on Overparameterized Linear Regressioncited by 3
  3. 2022Blessing of Nonconvexity in Deep Linear Models: Depth Flattens the Optimization Landscape Around the True Solutioncited by 3
  4. 2023Representation Learning Dynamics of Self-Supervised Modelscited by 3
  5. 2024Towards Understanding Epoch-wise Double descent in Two-layer Linear Neural Networkscited by 3
  6. 2025Diagonal Linear Networks and the Lasso Regularization Pathcited by 3
  7. 2024Improving Adaptivity via Over-Parameterization in Sequence Modelscited by 2
  8. 2026Never Saddle for Reparameterized Steepest Descent as Mirror Flowcited by 2
  9. 2025Understanding Incremental Learning with Closed-form Solution to Gradient Flow on Overparamerterized Matrix Factorizationcited by 1
  10. 2025Precise Dynamics of Diagonal Linear Networks: A Unifying Analysis by Dynamical Mean-Field Theorycited by 1
  11. 2023Sparse Mean Estimation in Adversarial Settings via Incremental Learningcited by 0
  12. 2024Tuning-Free Online Robust Principal Component Analysis through Implicit Regularizationcited by 0
  13. 2024Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theoriescited by 0
  14. 2025Tuning-Free Structured Sparse Recovery of Multiple Measurement Vectors using Implicit Regularizationcited by 0
  15. 2026Over-Alignment vs Over-Fitting: The Role of Feature Learning Strength in Generalizationcited by 0
  16. 2026Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learningcited by 0
  17. 2026HORST: Composing Optimizer Geometries for Sparse Transformer Trainingcited by 0
  18. 2026Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Ratecited by 0