Mitigating deep double descent by concatenating inputs
Not on the map. Why?
Abstract (arXiv)
The double descent curve is one of the most intriguing properties of deep neural networks. It contrasts the classical bias-variance curve with the behavior of modern neural networks, occurring where the number of samples nears the number of parameters. In this work, we explore the connection between the double descent phenomena and the number of samples in the deep neural network setting. In particular, we propose a construction which augments the existing dataset by artificially increasing the number of samples. This construction empirically mitigates the double descent curve in this setting. We reproduce existing work on deep double descent, and observe a smooth descent into the overparameterized region for our construction. This occurs both with respect to the model size, and with respect to the number epochs.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 25 of its 31 references that are in the map. First posted on arXiv 2 Jul 2021.
Builds on
25 of 31 references.
References · 25
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2016Densely Connected Convolutional Networkscited by 6,691
- 2016Wide Residual Networkscited by 3,057
- 2016Understanding deep learning requires rethinking generalizationcited by 1,657
- 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
- 2019Benign Overfitting in Linear Regressioncited by 544
- 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
- 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
- 2017High-dimensional dynamics of generalization error in neural networkscited by 238
- 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
- 2019Two models of double descent for weak featurescited by 194
- 2018To understand deep learning we need to understand kernel learningcited by 186
13 more
- 2019Linearized two-layers neural networks in high dimensioncited by 155
- 2019Harmless interpolation of noisy data in regressioncited by 121
- 2020Rethinking Bias-Variance Trade-off for Generalization of Neural Networkscited by 107
- 2020Optimal Regularization Can Mitigate Double Descentcited by 107
- 2019Scaling description of generalization with number of parameters in deep learningcited by 83
- 2018A Modern Take on the Bias-Variance Tradeoff in Neural Networkscited by 64
- 2020Triple descent and the two kinds of overfitting: Where & why do they appear?cited by 64
- 2018The jamming transition as a paradigm to understand the loss landscape of deep neural networkscited by 61
- 2020Multiple Descent: Design Your Own Generalization Curvecited by 44
- 2019More Data Can Hurt for Linear Regression: Sample-wise Double Descentcited by 37
- 2019Understanding overfitting peaks in generalization error: Analytical risk curves for l2 and l1 penalized interpolationcited by 23
- 2019A New Look at an Old Problem: A Universal Learning Approach to Linear Regressioncited by 11
- 2020A finite sample analysis of the benign overfitting phenomenon for ridge function estimationcited by 3
Built on by
Citing papers per year: 2023: 2; 2024: 1.
3 papers.