mlsanityData to 14 Oct 2026

Implicit Regularization Leads to Benign Overfitting for Sparse Linear Regression

arXiv 2302.00257 ↗First posted 1 Feb 2023cs.LG stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

In deep learning, often the training process finds an interpolator (a solution with 0 training loss), but the test loss is still low. This phenomenon, known as benign overfitting, is a major mystery that received a lot of recent attention. One common mechanism for benign overfitting is implicit regularization, where the training process leads to additional properties for the interpolator, often characterized by minimizing certain norms. However, even for a simple sparse linear regression problem y = β*top x +xi with sparse β^*, neither minimum ℓ1 or ℓ2 norm interpolator gives the optimal test loss. In this work, we give a different parametrization of the model which leads to a new implicit regularization effect that combines the benefit of ℓ1 and ℓ2 interpolators. We show that training our new model via gradient descent leads to an interpolator with near-optimal test loss. Our result is based on careful analysis of the training dynamics and provides another example of implicit regularization effect that goes beyond norm minimization.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 37 of its 50 references that are in the map. First posted on arXiv 1 Feb 2023.

Builds on

37 of 50 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  5. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  6. 2019Benign Overfitting in Linear Regressioncited by 549
  7. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  8. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  9. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  10. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  11. 2017High-dimensional dynamics of generalization error in neural networkscited by 241
  12. 2021Deep learning: a statistical viewpointcited by 240
25 more
  1. 2019Two models of double descent for weak featurescited by 195
  2. 2020Benign overfitting in ridge regressioncited by 153
  3. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
  4. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
  5. 2019Harmless interpolation of noisy data in regressioncited by 121
  6. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
  7. 2019Implicit Regularization for Optimal Sparse Recoverycited by 80
  8. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 75
  9. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 70
  10. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  11. 2021A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learningcited by 57
  12. 2021Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfittingcited by 54
  13. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  14. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  15. 2019In Defense of Uniform Convergence: Generalization via derandomization with an application to interpolating predictorscited by 44
  16. 2021Implicit Regularization in Tensor Factorizationcited by 44
  17. 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 39
  18. 2021Implicit Sparse Regularization: The Impact of Depth and Early Stoppingcited by 30
  19. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  20. 2020On Uniform Convergence and Low-Norm Interpolation Learningcited by 24
  21. 2021Foolish Crowds Support Benign Overfittingcited by 24
  22. 2019Understanding overfitting peaks in generalization error: Analytical risk curves for l2 and l1 penalized interpolationcited by 23
  23. 2021Understanding Deflation Process in Over-parametrized Tensor Decompositioncited by 20
  24. 2020On the robustness of minimum norm interpolators and regularized empirical risk minimizerscited by 19
  25. 2021Tight bounds for minimum l1-norm interpolation of noisy datacited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 0; 2026: 1.

4 papers.

  1. 2024Implicit Regularization of Gradient Flow on One-Layer Softmax Attentioncited by 19
  2. 2024FTMixer: Frequency and Time Domain Representations Fusion for Time Series Modelingcited by 4
  3. 2026Hard labels sampled from sparse targets mislead rotation invariant algorithmscited by 1
  4. 2024Implicit Regularization for Multi-label Feature Selectioncited by 0