Data to 14 Oct 2026

Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks

arXiv 1901.06523 ↗First posted 19 Jan 2019cs.LG stat.MLcited by 368

Not on the map. Why?

Abstract (arXiv)

We study the training process of Deep Neural Networks (DNNs) from the Fourier analysis perspective. We demonstrate a very universal Frequency Principle (F-Principle) -- DNNs often fit target functions from low to high frequencies -- on high-dimensional benchmark datasets such as MNIST/CIFAR10 and deep neural networks such as VGG16. This F-Principle of DNNs is opposite to the behavior of most conventional iterative numerical schemes (e.g., Jacobi method), which exhibit faster convergence for higher frequencies for various scientific computing problems. With a simple theory, we illustrate that this F-Principle results from the regularity of the commonly used activation functions. The F-Principle implies an implicit bias that DNNs tend to fit training data by a low-frequency function. This understanding provides an explanation of good generalization of DNNs on most real datasets and bad generalization of DNNs on parity function or randomized dataset.

Key facts. Cited by 368 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 34; 2022: 56; 2023: 53; 2024: 75; 2025: 68; 2026: 82.

368 papers.

Sort:

By area:

2022Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's nextcited by 608
2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 229
2023FreeU: Free Lunch in Diffusion U-Netcited by 196
2021Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equationscited by 145
2023An Expert's Guide to Training Physics-informed Neural Networkscited by 126
2022Diffusion Probabilistic Model Made Slimcited by 121
2021Improved architectures and training algorithms for deep operator networkscited by 86
2025Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Trainingcited by 81
2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
2022Overview frequency principle/spectral bias in deep learningcited by 69
2022HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imagingcited by 66
2021On the Frequency Bias of Generative Modelscited by 60

38 more

2024Convergence Analysis of Probability Flow ODE for Score-based Generative Modelscited by 57
2023Multi-stage Neural Networks: Function Approximator of Machine Precisioncited by 48
2023Spectrum-guided Multi-granularity Referring Video Object Segmentationcited by 48
2024Fredformer: Frequency Debiased Transformer for Time Series Forecastingcited by 45
2022Mitigating spectral bias for the multiscale operator learningcited by 44
2021Inverse-Dirichlet Weighting Enables Reliable Training of Physics Informed Neural Networkscited by 43
2024Neural Redshift: Random Networks are not Random Functionscited by 43
2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 38
2022Neural Implicit Flow: a mesh-agnostic dimensionality reduction paradigm of spatio-temporal datacited by 37
2024On the expressiveness and spectral bias of KANscited by 37
2021On Measuring and Controlling the Spectral Bias of the Deep Image Priorcited by 36
2022The Devil is in the Frequency: Geminated Gestalt Autoencoder for Self-Supervised Visual Pre-Trainingcited by 33
2021Towards Understanding the Condensation of Neural Networks at Initial Trainingcited by 32
2022A Multi-agent Reinforcement Learning Approach for Efficient Client Selection in Federated Learningcited by 31
2022Self-Supervised Coordinate Projection Network for Sparse-View Computed Tomographycited by 31
2024Neural Networks Learn Statistics of Increasing Complexitycited by 31
2025Mitigating Spectral Bias in Neural Operators via High-Frequency Scaling for Physical Systemscited by 31
2022On the Activation Function Dependence of the Spectral Bias of Neural Networkscited by 28
2022Castling-ViT: Compressing Self-Attention via Switching Towards Linear-Angular Attention at Vision Transformer Inferencecited by 28
2023What do neural networks learn in image classification? A frequency shortcut perspectivecited by 28
2021The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networkscited by 26
2022Implicit regularization of dropoutcited by 25
2022Incremental Spatial and Spectral Learning of Neural Operators for Solving Large-Scale PDEscited by 25
2024Toward a Better Understanding of Fourier Neural Operators from a Spectral Perspectivecited by 25
2021MOD-Net: A Machine Learning Approach via Model-Operator-Data Network for Solving PDEscited by 24
2023OceanNet: A principled neural operator-based digital twin for regional oceanscited by 24
2022Out-of-distribution Detection via Frequency-regularized Generative Modelscited by 23
2021Embedding Principle: a hierarchical structure of loss landscape of deep neural networkscited by 22
2022WaveGAN: Frequency-aware GAN for High-Fidelity Few-shot Image Generationcited by 22
2021Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networkscited by 21
2022Finite Expression Method for Solving High-Dimensional Partial Differential Equationscited by 21
2022Explaining the physics of transfer learning a data-driven subgrid-scale closure to a different turbulent flowcited by 20
2024SpeGCL: Self-supervised Graph Spectrum Contrastive Learning without Positive Samplescited by 20
2025Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformerscited by 20
2021Reproducing Activation Function for Deep Learningcited by 19
2021CPT: Efficient Deep Neural Network Training via Cyclic Precisioncited by 19
2022The Spectral Bias of Polynomial Neural Networkscited by 19
2022Hyper-parameter tuning of physics-informed neural networks: Application to Helmholtz problemscited by 19

50 most cited of 368 All 368 (JSON)