# Does Double Descent Occur in Self-Supervised Learning?

arXiv: https://arxiv.org/abs/2307.07872 · HTML: https://mlsanity.com/paper/2307.07872/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 13 references that are in the map.
- First posted on arXiv 15 Jul 2023.

## Builds on (most cited first, 9 of 9)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [A Modern Take on the Bias-Variance Tradeoff in Neural Networks](https://mlsanity.com/paper/1810.08591/) (2018)
- [High-dimensional analysis of double descent for linear regression with random projections](https://mlsanity.com/paper/2303.01372/) (2023)
- [More Data Can Hurt for Linear Regression: Sample-wise Double Descent](https://mlsanity.com/paper/1912.07242/) (2019)
- [Exact expressions for double descent and implicit regularization via surrogate random design](https://mlsanity.com/paper/1912.04533/) (2019)
- [Subspace Fitting Meets Regression: The Effects of Supervision and Orthonormality Constraints on Double Descent of Generalization Errors](https://mlsanity.com/paper/2002.10614/) (2020)
- [Double Descent and Other Interpolation Phenomena in GANs](https://mlsanity.com/paper/2106.04003/) (2021)

## Built on by (most cited first, 3 of 3)

- [Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representations](https://mlsanity.com/paper/2406.09366/) (2024)
- [Unveiling Multiple Descents in Unsupervised Autoencoders](https://mlsanity.com/paper/2406.11703/) (2024)
- [Impact of Bottleneck Layers and Skip Connections on the Generalization of Linear Denoising Autoencoders](https://mlsanity.com/paper/2505.24668/) (2025)

## Abstract (arXiv)

Most investigations into double descent have focused on supervised models while the few works studying self-supervised settings find a surprising lack of the phenomenon. These results imply that double descent may not exist in self-supervised models. We show this empirically using a standard and linear autoencoder, two previously unstudied settings. The test loss is found to have either a classical U-shape or to monotonically decrease instead of exhibiting a double-descent curve. We hope that further work on this will help elucidate the theoretical underpinnings of this phenomenon.