mlsanityData to 7 Oct 2026

Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networks

arXiv 2102.00200 ↗First posted 30 Jan 2021cs.LG physics.data-ancited by 14

Not on the map. Why?

Abstract (arXiv)

Why heavily parameterized neural networks (NNs) do not overfit the data is an important long standing open question. We propose a phenomenological model of the NN training to explain this non-overfitting puzzle. Our linear frequency principle (LFP) model accounts for a key dynamical feature of NNs: they learn low frequencies first, irrespective of microscopic details. Theory based on our LFP model shows that low frequency dominance of target functions is the key condition for the non-overfitting of NNs and is verified by experiments. Furthermore, through an ideal two-layer NN, we unravel how detailed microscopic NN training dynamics statistically gives rise to a LFP model with quantitative prediction power.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 11 of its 48 references that are in the map. First posted on arXiv 30 Jan 2021.

Builds on

11 of 48 references.

References · 11

  1. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  2. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 360
  3. 2017Deep Potential Molecular Dynamics: a scalable model with the accuracy of quantum mechanicscited by 213
  4. 2018A high-bias, low-variance introduction to Machine Learning for physicistscited by 108
  5. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  6. 2020Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domainscited by 78
  7. 2018Quantum Entanglement in Deep Learning Architecturescited by 49
  8. 2019A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamicscited by 49
  9. 2017Artificial Intelligence and Statisticscited by 43
  10. 2016A Convolutional Neural Network Neutrino Event Classifiercited by 26
  11. 2019Frequency-Aware Reconstruction of Fluid Simulations with Generative Networkscited by 7

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 6; 2023: 1; 2024: 3; 2025: 1.

14 papers.

Sort:

  1. 2022Overview frequency principle/spectral bias in deep learningcited by 68
  2. 2022Implicit regularization of dropoutcited by 24
  3. 2021Embedding Principle: a hierarchical structure of loss landscape of deep neural networkscited by 21
  4. 2023Loss Spike in Training Neural Networkscited by 15
  5. 2021Subspace Decomposition based DNN algorithm for elliptic type multi-scale PDEscited by 10
  6. 2024Anchor function: a type of benchmark functions for studying language modelscited by 10
  7. 2022Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networkscited by 7
  8. 2022Linear Stability Hypothesis and Rank Stratification for Nonlinear Modelscited by 7
  9. 2024Understanding the dynamics of the frequency bias in neural networkscited by 4
  10. 2024A rationale from frequency perspective for grokking in training neural networkcited by 3
  11. 2021Dropout in Training Neural Networks: Flatness of Solution and Noise Structurecited by 2
  12. 2025Hypernym Bias: Unraveling Deep Classifier Training Dynamics through the Lens of Class Hierarchycited by 2
2 more
  1. 2022Effective Dynamics of Generative Adversarial Networkscited by 1
  2. 2022Limitation of Characterizing Implicit Regularization by Data-independent Functionscited by 0