mlsanityData to 7 Oct 2026

Regularization-wise double descent: Why it occurs and how to eliminate it

arXiv 2206.01378 ↗First posted 3 Jun 2022cs.LG stat.MLcited by 9

Not on the map. Why?

Abstract (arXiv)

The risk of overparameterized models, in particular deep neural networks, is often double-descent shaped as a function of the model size. Recently, it was shown that the risk as a function of the early-stopping time can also be double-descent shaped, and this behavior can be explained as a super-position of bias-variance tradeoffs. In this paper, we show that the risk of explicit L2-regularized models can exhibit double descent behavior as a function of the regularization strength, both in theory and practice. We find that for linear regression, a double descent shaped risk is caused by a superposition of bias-variance tradeoffs corresponding to different parts of the model and can be mitigated by scaling the regularization strength of each part appropriately. Motivated by this result, we study a two-layer neural network and show that double descent can be eliminated by adjusting the regularization strengths for the first and second layer. Lastly, we study a 5-layer CNN and ResNet-18 trained on CIFAR-10 with label noise, and CIFAR-100 without label noise, and demonstrate that all exhibit double descent behavior as a function of the regularization strength.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 25 of its 31 references that are in the map. First posted on arXiv 3 Jun 2022.

Builds on

25 of 31 references.

References · 25

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2017A Closer Look at Memorization in Deep Networkscited by 936
  4. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  5. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  6. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  7. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  8. 2019Two models of double descent for weak featurescited by 194
  9. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  10. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  11. 2020Rethinking Bias-Variance Trade-off for Generalization of Neural Networkscited by 107
  12. 2020Optimal Regularization Can Mitigate Double Descentcited by 107
13 more
  1. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  2. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
  3. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  4. 2020Understanding Double Descent Requires a Fine-Grained Bias-Variance Decompositioncited by 75
  5. 2018A Continuous-Time View of Early Stopping for Least Squarescited by 67
  6. 2018A Modern Take on the Bias-Variance Tradeoff in Neural Networkscited by 64
  7. 2020Implicit Regularization of Random Feature Modelscited by 54
  8. 2020Early Stopping in Deep Networks: Double Descent and How to Eliminate itcited by 40
  9. 2020On the training dynamics of deep networks with L2 regularizationcited by 38
  10. 2019More Data Can Hurt for Linear Regression: Sample-wise Double Descentcited by 37
  11. 2019Understanding overfitting peaks in generalization error: Analytical risk curves for l2 and l1 penalized interpolationcited by 23
  12. 2021Optimization Variance: Exploring Generalization Properties of DNNscited by 4
  13. 2019Image recognition from raw labels collected without annotatorscited by 1

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 5; 2024: 3.

9 papers.

  1. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 142
  2. 2024On Regularization via Early Stopping for Least Squares Regressioncited by 10
  3. 2024Survival of the Fittest Representation: A Case Study with Modular Additioncited by 7
  4. 2023DSD2: Can We Dodge Sparse Double Descent and Compress the Neural Network Worry-Free?cited by 6
  5. 2023Least Squares Regression Can Exhibit Under-Parameterized Double Descentcited by 4
  6. 2023Can we avoid Double Descent in Deep Neural Networks?cited by 2
  7. 2023Sparse Double Descent in Vision Transformers: real or phantom threat?cited by 2
  8. 2023The Quest of Finding the Antidote to Sparse Double Descentcited by 0
  9. 2024eGAD! double descent is explained by Generalized Aliasing Decompositioncited by 0