# Rethink the Connections among Generalization, Memorization and the Spectral Bias of DNNs

arXiv: https://arxiv.org/abs/2004.13954 · HTML: https://mlsanity.com/paper/2004.13954/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2020.

## Built on by (most cited first, 11 of 11)

- [Spectral Bias in Practice: The Role of Function Frequency in Generalization](https://mlsanity.com/paper/2110.02424/) (2021)
- [On the Activation Function Dependence of the Spectral Bias of Neural Networks](https://mlsanity.com/paper/2208.04924/) (2022)
- [Face.evoLVe: A High-Performance Face Recognition Library](https://mlsanity.com/paper/2107.08621/) (2021)
- [Multi-scale Feature Learning Dynamics: Insights for Double Descent](https://mlsanity.com/paper/2112.03215/) (2021)
- [Mitigating Privacy Risk in Membership Inference by Convex-Concave Loss](https://mlsanity.com/paper/2402.05453/) (2024)
- [Lazy vs hasty: linearization in deep networks impacts learning schedule based on example difficulty](https://mlsanity.com/paper/2209.09658/) (2022)
- [Optimization Variance: Exploring Generalization Properties of DNNs](https://mlsanity.com/paper/2106.01714/) (2021)
- [Defending Membership Inference Attacks via Privacy-aware Sparsity Tuning](https://mlsanity.com/paper/2410.06814/) (2024)
- [Leveraging Unlabeled Data to Track Memorization](https://mlsanity.com/paper/2212.04461/) (2022)
- [Hypernym Bias: Unraveling Deep Classifier Training Dynamics through the Lens of Class Hierarchy](https://mlsanity.com/paper/2502.12125/) (2025)
- [Singularity Formation: Synergy in Theoretical, Numerical and Machine Learning Approaches](https://mlsanity.com/paper/2604.16842/) (2026)

## Abstract (arXiv)

Over-parameterized deep neural networks (DNNs) with sufficient capacity to memorize random noise can achieve excellent generalization performance, challenging the bias-variance trade-off in classical learning theory. Recent studies claimed that DNNs first learn simple patterns and then memorize noise; some other works showed a phenomenon that DNNs have a spectral bias to learn target functions from low to high frequencies during training. However, we show that the monotonicity of the learning bias does not always hold: under the experimental setup of deep double descent, the high-frequency components of DNNs diminish in the late stage of training, leading to the second descent of the test error. Besides, we find that the spectrum of DNNs can be applied to indicating the second descent of the test error, even though it is calculated from the training set only.