# Understanding training and generalization in deep learning by Fourier analysis

arXiv: https://arxiv.org/abs/1808.04295 · HTML: https://mlsanity.com/paper/1808.04295/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Aug 2018.

## Built on by (most cited first, 30 of 44)

- [Overview frequency principle/spectral bias in deep learning](https://mlsanity.com/paper/2201.07395/) (2022)
- [Geometry-Consistent Neural Shape Representation with Implicit Displacement Fields](https://mlsanity.com/paper/2106.05187/) (2021)
- [Neural networks trained with SGD learn distributions of increasing complexity](https://mlsanity.com/paper/2211.11567/) (2022)
- [Adaptive Frequency Filters As Efficient Global Token Mixers](https://mlsanity.com/paper/2307.14008/) (2023)
- [Implicit Regularization in Tensor Factorization](https://mlsanity.com/paper/2102.09972/) (2021)
- [Inverse-Dirichlet Weighting Enables Reliable Training of Physics Informed Neural Networks](https://mlsanity.com/paper/2107.00940/) (2021)
- [Deep Frequency Filtering for Domain Generalization](https://mlsanity.com/paper/2203.12198/) (2022)
- [Discovering and Explaining the Representation Bottleneck of DNNs](https://mlsanity.com/paper/2111.06236/) (2021)
- [Improved Implicit Neural Representation with Fourier Reparameterized Training](https://mlsanity.com/paper/2401.07402/) (2024)
- [On the Activation Function Dependence of the Spectral Bias of Neural Networks](https://mlsanity.com/paper/2208.04924/) (2022)
- [The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networks](https://mlsanity.com/paper/2110.03922/) (2021)
- [SpeGCL: Self-supervised Graph Spectrum Contrastive Learning without Positive Samples](https://mlsanity.com/paper/2410.10365/) (2024)
- [MLP Can Be A Good Transformer Learner](https://mlsanity.com/paper/2404.05657/) (2024)
- [FreSh: Frequency Shifting for Accelerated Neural Representation Learning](https://mlsanity.com/paper/2410.05050/) (2024)
- [Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learning](https://mlsanity.com/paper/2410.16105/) (2024)
- [Exploring The Effect of High-frequency Components in GANs Training](https://mlsanity.com/paper/2103.11093/) (2021)
- [Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Training](https://mlsanity.com/paper/2405.18296/) (2024)
- [Inductive Gradient Adjustment For Spectral Bias In Implicit Neural Representations](https://mlsanity.com/paper/2410.13271/) (2024)
- [Linear CNNs Discover the Statistical Structure of the Dataset Using Only the Most Dominant Frequencies](https://mlsanity.com/paper/2303.02034/) (2023)
- [Exploring the Low-Pass Filtering Behavior in Image Super-Resolution](https://mlsanity.com/paper/2405.07919/) (2024)
- [Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithm](https://mlsanity.com/paper/2207.04612/) (2022)
- [How You Start Matters for Generalization](https://mlsanity.com/paper/2206.08558/) (2022)
- [Implicit Neural Representations with Fourier Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2409.09323/) (2024)
- [Frequency-adaptive Multi-scale Deep Neural Networks](https://mlsanity.com/paper/2410.00053/) (2024)
- [Spectral Bias in Variational Quantum Machine Learning](https://mlsanity.com/paper/2506.22555/) (2025)
- [An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network](https://mlsanity.com/paper/2105.11675/) (2021)
- [Trap of Feature Diversity in the Learning of MLPs](https://mlsanity.com/paper/2112.00980/) (2021)
- [Understanding the dynamics of the frequency bias in neural networks](https://mlsanity.com/paper/2405.14957/) (2024)
- [PDEfuncta: Spectrally-Aware Neural Representation for PDE Solution Modeling](https://mlsanity.com/paper/2506.12790/) (2025)
- [From Tables to Signals: Revealing Spectral Adaptivity in TabPFN](https://mlsanity.com/paper/2511.18278/) (2025)

## Abstract (arXiv)

Background: It is still an open research area to theoretically understand why Deep Neural Networks (DNNs)---equipped with many more parameters than training data and trained by (stochastic) gradient-based methods---often achieve remarkably low generalization error. Contribution: We study DNN training by Fourier analysis. Our theoretical framework explains: i) DNN with (stochastic) gradient-based methods often endows low-frequency components of the target function with a higher priority during the training; ii) Small initialization leads to good generalization ability of DNN while preserving the DNN's ability to fit any function. These results are further confirmed by experiments of DNNs fitting the following datasets, that is, natural images, one-dimensional functions and MNIST dataset.