mlsanityData to 7 Oct 2026

The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networks

arXiv 2108.11489 ↗First posted 25 Aug 2021stat.ML cs.LG math.STcited by 18

Not on the map. Why?

Abstract (arXiv)

The recent success of neural network models has shone light on a rather surprising statistical phenomenon: statistical models that perfectly fit noisy data can generalize well to unseen test data. Understanding this phenomenon of textitbenign overfitting has attracted intense theoretical and empirical study. In this paper, we consider interpolating two-layer linear neural networks trained with gradient flow on the squared loss and derive bounds on the excess risk when the covariates satisfy sub-Gaussianity and anti-concentration properties, and the noise is independent and sub-Gaussian. By leveraging recent results that characterize the implicit bias of this estimator, our bounds emphasize the role of both the quality of the initialization as well as the properties of the data covariance matrix in achieving low excess risk.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 37 of its 50 references that are in the map. First posted on arXiv 25 Aug 2021.

Builds on

37 of 50 references.

References · 37

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  4. 2019Benign Overfitting in Linear Regressioncited by 544
  5. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  6. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  7. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  8. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  9. 2017Implicit Regularization in Matrix Factorizationcited by 318
  10. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  11. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  12. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
25 more
  1. 2021Deep learning: a statistical viewpointcited by 239
  2. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  3. 2020Benign overfitting in ridge regressioncited by 152
  4. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  5. 2019Harmless interpolation of noisy data in regressioncited by 121
  6. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  7. 2020Rethinking Bias-Variance Trade-off for Generalization of Neural Networkscited by 107
  8. 2018Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolatecited by 99
  9. 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
  10. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
  11. 2019A Model of Double Descent for High-dimensional Binary Linear Classificationcited by 81
  12. 2020Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regimecited by 81
  13. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 74
  14. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  15. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  16. 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 63
  17. 2021Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfittingcited by 54
  18. 2018Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularizationcited by 52
  19. 2019In Defense of Uniform Convergence: Generalization via derandomization with an application to interpolating predictorscited by 44
  20. 2020A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-ℓ1-Norm Interpolated Classifierscited by 44
  21. 2020On the proliferation of support vectors in high dimensionscited by 32
  22. 2021Towards an Understanding of Benign Overfitting in Neural Networkscited by 29
  23. 2020On the robustness of minimum norm interpolators and regularized empirical risk minimizerscited by 18
  24. 2021Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Normcited by 18
  25. 2020Interpolating Predictors in High-Dimensional Factor Regressioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 5; 2024: 5; 2025: 1; 2026: 1.

18 papers.

Sort:

  1. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  2. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  3. 2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
  4. 2023Noisy Interpolation Learning with Shallow Univariate ReLU Networkscited by 11
  5. 2023Benign Overfitting in Deep Neural Networks under Lazy Trainingcited by 10
  6. 2022Learning Ability of Interpolating Deep Convolutional Neural Networkscited by 9
  7. 2022Deep Linear Networks can Benignly Overfit when Shallow Ones Docited by 8
  8. 2024Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionalitycited by 7
  9. 2022From Gradient Flow on Population Loss to Learning with Stochastic Gradient Descentcited by 5
  10. 2024Provable Tempered Overfitting of Minimal Nets and Typical Netscited by 5
  11. 2023General Loss Functions Lead to (Approximate) Interpolation in High Dimensionscited by 4
  12. 2022Investigating the Impact of Model Width and Density on Generalization in Presence of Label Noisecited by 2
6 more
  1. 2023Representation Transfer Learning via Multiple Pre-trained models for Linear Regressioncited by 2
  2. 2024Bayesian Inference for Consistent Predictions in Overparameterized Nonlinear Regressioncited by 0
  3. 2024Benign Overfitting under Learning Rate Conditions for α Sub-exponential Inputcited by 0
  4. 2024Implicit Regularization for Multi-label Feature Selectioncited by 0
  5. 2025The Interplay of Statistics and Noisy Optimization: Learning Linear Predictors with Random Data Weightscited by 0
  6. 2026A mathematical theory of balancing relational generalization and memorizationcited by 0