# The Slow Deterioration of the Generalization Error of the Random Feature Model

arXiv: https://arxiv.org/abs/2008.05621 · HTML: https://mlsanity.com/paper/2008.05621/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Aug 2020.

## Built on by (most cited first, 8 of 8)

- [Overview frequency principle/spectral bias in deep learning](https://mlsanity.com/paper/2201.07395/) (2022)
- [Towards Understanding the Condensation of Neural Networks at Initial Training](https://mlsanity.com/paper/2105.11686/) (2021)
- [Subspace Decomposition based DNN algorithm for elliptic type multi-scale PDEs](https://mlsanity.com/paper/2112.06660/) (2021)
- [Conditioning of Random Feature Matrices: Double Descent and Generalization Error](https://mlsanity.com/paper/2110.11477/) (2021)
- [A Mathematical Framework for Learning Probability Distributions](https://mlsanity.com/paper/2212.11481/) (2022)
- [Side Effects of Learning from Low-dimensional Data Embedded in a Euclidean Space](https://mlsanity.com/paper/2203.00614/) (2022)
- [Frequency Principle in Deep Learning Beyond Gradient-descent-based Training](https://mlsanity.com/paper/2101.00747/) (2021)
- [The double descent and Runge phenomena in overparametrized polynomial interpolation](https://mlsanity.com/paper/2609.37657/) (2026)

## Abstract (arXiv)

The random feature model exhibits a kind of resonance behavior when the number of parameters is close to the training sample size. This behavior is characterized by the appearance of large generalization gap, and is due to the occurrence of very small eigenvalues for the associated Gram matrix. In this paper, we examine the dynamic behavior of the gradient descent algorithm in this regime. We show, both theoretically and experimentally, that there is a dynamic self-correction mechanism at work: The larger the eventual generalization gap, the slower it develops, both because of the small eigenvalues. This gives us ample time to stop the training process and obtain solutions with good generalization property.