# Theory of the Frequency Principle for General Deep Neural Networks

arXiv: https://arxiv.org/abs/1906.09235 · HTML: https://mlsanity.com/paper/1906.09235/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2019.

## Built on by (most cited first, 30 of 46)

- [Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks](https://mlsanity.com/paper/2006.13198/) (2020)
- [Overview frequency principle/spectral bias in deep learning](https://mlsanity.com/paper/2201.07395/) (2022)
- [Image Shortcut Squeezing: Countering Perturbative Availability Poisons with Compression](https://mlsanity.com/paper/2301.13838/) (2023)
- [Implicit regularization of dropout](https://mlsanity.com/paper/2207.05952/) (2022)
- [Embedding Principle: a hierarchical structure of loss landscape of deep neural networks](https://mlsanity.com/paper/2111.15527/) (2021)
- [On Numerical Integration in Neural Ordinary Differential Equations](https://mlsanity.com/paper/2206.07335/) (2022)
- [Reproducing Activation Function for Deep Learning](https://mlsanity.com/paper/2101.04844/) (2021)
- [No Time to Train: Empowering Non-Parametric Networks for Few-shot 3D Scene Segmentation](https://mlsanity.com/paper/2404.04050/) (2024)
- [Loss Spike in Training Neural Networks](https://mlsanity.com/paper/2305.12133/) (2023)
- [Learning Gabor Texture Features for Fine-Grained Recognition](https://mlsanity.com/paper/2308.05396/) (2023)
- [The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimation](https://mlsanity.com/paper/2103.11488/) (2021)
- [Machine learning and domain decomposition methods -- a survey](https://mlsanity.com/paper/2312.14050/) (2023)
- [Regularize implicit neural representation by itself](https://mlsanity.com/paper/2303.15484/) (2023)
- [Anchor function: a type of benchmark functions for studying language models](https://mlsanity.com/paper/2401.08309/) (2024)
- [Simplicity Bias of Two-Layer Networks beyond Linearly Separable Data](https://mlsanity.com/paper/2405.17299/) (2024)
- [Subspace Decomposition based DNN algorithm for elliptic type multi-scale PDEs](https://mlsanity.com/paper/2112.06660/) (2021)
- [Defects of Convolutional Decoder Networks in Frequency Representation](https://mlsanity.com/paper/2210.09020/) (2022)
- [FreqKV: Key-Value Compression in Frequency Domain for Context Window Extension](https://mlsanity.com/paper/2505.00570/) (2025)
- [Spectral-DP: Differentially Private Deep Learning through Spectral Perturbation and Filtering](https://mlsanity.com/paper/2307.13231/) (2023)
- [Towards Inadequately Pre-trained Models in Transfer Learning](https://mlsanity.com/paper/2203.04668/) (2022)
- [How You Start Matters for Generalization](https://mlsanity.com/paper/2206.08558/) (2022)
- [Invisible Backdoor Attacks Using Data Poisoning in the Frequency Domain](https://mlsanity.com/paper/2207.04209/) (2022)
- [Frequency Principle in Deep Learning Beyond Gradient-descent-based Training](https://mlsanity.com/paper/2101.00747/) (2021)
- [An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network](https://mlsanity.com/paper/2105.11675/) (2021)
- [Understanding the dynamics of the frequency bias in neural networks](https://mlsanity.com/paper/2405.14957/) (2024)
- [Wavelet Regularization Benefits Adversarial Training](https://mlsanity.com/paper/2206.03727/) (2022)
- [A rationale from frequency perspective for grokking in training neural network](https://mlsanity.com/paper/2405.17479/) (2024)
- [Spatiotemporal wall pressure forecast of a rectangular cylinder with physics-aware DeepU-Fourier neural network](https://mlsanity.com/paper/2508.03183/) (2025)
- [Multigrade Neural Network Approximation](https://mlsanity.com/paper/2601.16884/) (2026)
- [Layer-wise Geometric Approximation Rates for Deep Networks](https://mlsanity.com/paper/2604.20219/) (2026)

## Abstract (arXiv)

Along with fruitful applications of Deep Neural Networks (DNNs) to realistic problems, recently, some empirical studies of DNNs reported a universal phenomenon of Frequency Principle (F-Principle): a DNN tends to learn a target function from low to high frequencies during the training. The F-Principle has been very useful in providing both qualitative and quantitative understandings of DNNs. In this paper, we rigorously investigate the F-Principle for the training dynamics of a general DNN at three stages: initial stage, intermediate stage, and final stage. For each stage, a theorem is provided in terms of proper quantities characterizing the F-Principle. Our results are general in the sense that they work for multilayer networks with general activation functions, population densities of data, and a large class of loss functions. Our work lays a theoretical foundation of the F-Principle for a better understanding of the training process of DNNs.