# Can we avoid Double Descent in Deep Neural Networks?

arXiv: https://arxiv.org/abs/2302.13259 · HTML: https://mlsanity.com/paper/2302.13259/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 21 references that are in the map.
- First posted on arXiv 26 Feb 2023.

## Builds on (most cited first, 18 of 18)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [Dimensionality-Driven Learning with Noisy Labels](https://mlsanity.com/paper/1806.02612/) (2018)
- [Multi-task self-supervised learning for Robust Speech Recognition](https://mlsanity.com/paper/2001.09239/) (2020)
- [Harmless interpolation of noisy data in regression](https://mlsanity.com/paper/1903.09139/) (2019)
- [Optimal Regularization Can Mitigate Double Descent](https://mlsanity.com/paper/2003.01897/) (2020)
- [A jamming transition from under- to over-parametrization affects loss landscape and generalization](https://mlsanity.com/paper/1810.09665/) (2018)
- [The jamming transition as a paradigm to understand the loss landscape of deep neural networks](https://mlsanity.com/paper/1809.09349/) (2018)
- [Provable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networks](https://mlsanity.com/paper/2012.08749/) (2020)
- [Sparse Double Descent: Where Network Pruning Aggravates Overfitting](https://mlsanity.com/paper/2206.08684/) (2022)
- [To update or not to update? Neurons at equilibrium in deep models](https://mlsanity.com/paper/2207.09455/) (2022)
- [Multiple Descent in the Multiple Random Feature Model](https://mlsanity.com/paper/2208.09897/) (2022)
- [Regularization-wise double descent: Why it occurs and how to eliminate it](https://mlsanity.com/paper/2206.01378/) (2022)
- [The rise of the lottery heroes: why zero-shot pruning is hard](https://mlsanity.com/paper/2202.12400/) (2022)

## Built on by (most cited first, 2 of 2)

- [Sparse Double Descent in Vision Transformers: real or phantom threat?](https://mlsanity.com/paper/2307.14253/) (2023)
- [The Quest of Finding the Antidote to Sparse Double Descent](https://mlsanity.com/paper/2308.16596/) (2023)

## Abstract (arXiv)

Finding the optimal size of deep learning models is very actual and of broad impact, especially in energy-saving schemes. Very recently, an unexpected phenomenon, the ``double descent'', has caught the attention of the deep learning community. As the model's size grows, the performance gets first worse, and then goes back to improving. It raises serious questions about the optimal model's size to maintain high generalization: the model needs to be sufficiently over-parametrized, but adding too many parameters wastes training resources. Is it possible to find, in an efficient way, the best trade-off? Our work shows that the double descent phenomenon is potentially avoidable with proper conditioning of the learning problem, but a final answer is yet to be found. We empirically observe that there is hope to dodge the double descent in complex scenarios with proper regularization, as a simple ℓ2 regularization is already positively contributing to such a perspective.