mlsanityData to 7 Oct 2026

Deep Linear Networks can Benignly Overfit when Shallow Ones Do

arXiv 2209.09315 ↗First posted 19 Sep 2022cs.LG cs.AI math.STcited by 8

Not on the map. Why?

Abstract (arXiv)

We bound the excess risk of interpolating deep linear networks trained using gradient flow. In a setting previously used to establish risk bounds for the minimum ℓ2-norm interpolant, we show that randomly initialized deep linear networks can closely approximate or even match known bounds for the minimum ℓ2-norm interpolant. Our analysis also reveals that interpolating deep linear models have exactly the same conditional variance as the minimum ℓ2-norm solution. Since the noise affects the excess risk only through the conditional variance, this implies that depth does not improve the algorithm's ability to "hide the noise". Our simulations verify that aspects of our bounds reflect typical behavior for simple data distributions. We also find that similar phenomena are seen in simulations with ReLU networks, although the situation there is more nuanced.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 61 of its 75 references that are in the map. First posted on arXiv 19 Sep 2022.

Builds on

61 of 75 references.

References · 61

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  4. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  5. 2019Benign Overfitting in Linear Regressioncited by 544
  6. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  7. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  8. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  9. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  10. 2017Implicit Regularization in Matrix Factorizationcited by 318
  11. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  12. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
38 more
  1. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  2. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  3. 2021Deep learning: a statistical viewpointcited by 239
  4. 2016Deep Learning without Poor Local Minimacited by 233
  5. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  6. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  7. 2020Benign overfitting in ridge regressioncited by 152
  8. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  9. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  10. 2019Harmless interpolation of noisy data in regressioncited by 121
  11. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  12. 2018Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolatecited by 99
  13. 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
  14. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
  15. 2018Does data interpolation contradict statistical optimality?cited by 95
  16. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  17. 2019A Model of Double Descent for High-dimensional Binary Linear Classificationcited by 81
  18. 2020Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regimecited by 81
  19. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 74
  20. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  21. 2020Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networkscited by 73
  22. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  23. 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 63
  24. 2021Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfittingcited by 54
  25. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  26. 2018Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularizationcited by 52
  27. 2019Width Provably Matters in Optimization for Deep Linear Neural Networkscited by 52
  28. 2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 48
  29. 2020Implicit Regularization in ReLU Networks with the Square Losscited by 45
  30. 2018Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networkscited by 44
  31. 2019In Defense of Uniform Convergence: Generalization via derandomization with an application to interpolating predictorscited by 44
  32. 2020A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-ℓ1-Norm Interpolated Classifierscited by 44
  33. 2022The Implicit Bias of Benign Overfittingcited by 38
  34. 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 38
  35. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  36. 2020On the proliferation of support vectors in high dimensionscited by 32
  37. 2020On the Global Convergence of Training Deep Linear ResNetscited by 29
  38. 2021Towards an Understanding of Benign Overfitting in Neural Networkscited by 29

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 6; 2024: 1; 2025: 1.

8 papers.

  1. 2023Benign Overfitting for Two-layer ReLU Convolutional Neural Networkscited by 34
  2. 2023Why Does Sharpness-Aware Minimization Generalize Better Than SGD?cited by 27
  3. 2023From Tempered to Benign Overfitting in ReLU Neural Networkscited by 16
  4. 2023Benign Overfitting in Two-Layer ReLU Convolutional Neural Networks for XOR Datacited by 10
  5. 2023Precise Asymptotic Generalization for Multiclass Classification with Overparameterized Linear Modelscited by 3
  6. 2023Efficient Compression of Overparameterized Deep Models through Low-Dimensional Learning Dynamicscited by 3
  7. 2024Towards Understanding Epoch-wise Double descent in Two-layer Linear Neural Networkscited by 3
  8. 2025Understanding Private Learning From Feature Perspectivecited by 2