# Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networks

arXiv: https://arxiv.org/abs/2102.00200 · HTML: https://mlsanity.com/paper/2102.00200/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 48 references that are in the map.
- First posted on arXiv 30 Jan 2021.

## Builds on (most cited first, 11 of 11)

- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks](https://mlsanity.com/paper/1901.06523/) (2019)
- [Deep Potential Molecular Dynamics: a scalable model with the accuracy of quantum mechanics](https://mlsanity.com/paper/1707.09571/) (2017)
- [A high-bias, low-variance introduction to Machine Learning for physicists](https://mlsanity.com/paper/1803.08823/) (2018)
- [Mean Field Analysis of Neural Networks: A Central Limit Theorem](https://mlsanity.com/paper/1808.09372/) (2018)
- [Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains](https://mlsanity.com/paper/2007.11207/) (2020)
- [Quantum Entanglement in Deep Learning Architectures](https://mlsanity.com/paper/1803.09780/) (2018)
- [A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics](https://mlsanity.com/paper/1904.04326/) (2019)
- [Artificial Intelligence and Statistics](https://mlsanity.com/paper/1712.03779/) (2017)
- [A Convolutional Neural Network Neutrino Event Classifier](https://mlsanity.com/paper/1604.01444/) (2016)
- [Frequency-Aware Reconstruction of Fluid Simulations with Generative Networks](https://mlsanity.com/paper/1912.08776/) (2019)

## Built on by (most cited first, 14 of 14)

- [Overview frequency principle/spectral bias in deep learning](https://mlsanity.com/paper/2201.07395/) (2022)
- [Implicit regularization of dropout](https://mlsanity.com/paper/2207.05952/) (2022)
- [Embedding Principle: a hierarchical structure of loss landscape of deep neural networks](https://mlsanity.com/paper/2111.15527/) (2021)
- [Loss Spike in Training Neural Networks](https://mlsanity.com/paper/2305.12133/) (2023)
- [Anchor function: a type of benchmark functions for studying language models](https://mlsanity.com/paper/2401.08309/) (2024)
- [Subspace Decomposition based DNN algorithm for elliptic type multi-scale PDEs](https://mlsanity.com/paper/2112.06660/) (2021)
- [Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networks](https://mlsanity.com/paper/2205.13283/) (2022)
- [Linear Stability Hypothesis and Rank Stratification for Nonlinear Models](https://mlsanity.com/paper/2211.11623/) (2022)
- [Understanding the dynamics of the frequency bias in neural networks](https://mlsanity.com/paper/2405.14957/) (2024)
- [A rationale from frequency perspective for grokking in training neural network](https://mlsanity.com/paper/2405.17479/) (2024)
- [Dropout in Training Neural Networks: Flatness of Solution and Noise Structure](https://mlsanity.com/paper/2111.01022/) (2021)
- [Hypernym Bias: Unraveling Deep Classifier Training Dynamics through the Lens of Class Hierarchy](https://mlsanity.com/paper/2502.12125/) (2025)
- [Effective Dynamics of Generative Adversarial Networks](https://mlsanity.com/paper/2212.04580/) (2022)
- [Limitation of Characterizing Implicit Regularization by Data-independent Functions](https://mlsanity.com/paper/2201.12198/) (2022)

## Abstract (arXiv)

Why heavily parameterized neural networks (NNs) do not overfit the data is an important long standing open question. We propose a phenomenological model of the NN training to explain this non-overfitting puzzle. Our linear frequency principle (LFP) model accounts for a key dynamical feature of NNs: they learn low frequencies first, irrespective of microscopic details. Theory based on our LFP model shows that low frequency dominance of target functions is the key condition for the non-overfitting of NNs and is verified by experiments. Furthermore, through an ideal two-layer NN, we unravel how detailed microscopic NN training dynamics statistically gives rise to a LFP model with quantitative prediction power.