# Overview frequency principle/spectral bias in deep learning

arXiv: https://arxiv.org/abs/2201.07395 · HTML: https://mlsanity.com/paper/2201.07395/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- Builds on 90 of its 135 references that are in the map.
- First posted on arXiv 19 Jan 2022.

## Builds on (most cited first, 30 of 90)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [DeepXDE: A deep learning library for solving differential equations](https://mlsanity.com/paper/1907.04502/) (2019)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks](https://mlsanity.com/paper/1901.08584/) (2019)
- [Fantastic Generalization Measures and Where to Find Them](https://mlsanity.com/paper/1912.02178/) (2019)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport](https://mlsanity.com/paper/1805.09545/) (2018)
- [Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks](https://mlsanity.com/paper/1901.06523/) (2019)
- [Neural Additive Models: Interpretable Machine Learning with Neural Nets](https://mlsanity.com/paper/2004.13912/) (2020)
- [On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks](https://mlsanity.com/paper/2012.10047/) (2020)
- [The Loss Surfaces of Multilayer Networks](https://mlsanity.com/paper/1412.0233/) (2014)
- [Learned Initializations for Optimizing Coordinate-Based Neural Representations](https://mlsanity.com/paper/2012.02189/) (2020)
- [SGD on Neural Networks Learns Functions of Increasing Complexity](https://mlsanity.com/paper/1905.11604/) (2019)
- [Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks](https://mlsanity.com/paper/2002.02561/) (2020)
- [Neural Reflectance Fields for Appearance Acquisition](https://mlsanity.com/paper/2008.03824/) (2020)
- [Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations](https://mlsanity.com/paper/1706.04702/) (2017)

## Built on by (most cited first, 30 of 69)

- [WebSailor: Navigating Super-human Reasoning for Web Agent](https://mlsanity.com/paper/2507.02592/) (2025)
- [The Devil is in the Frequency: Geminated Gestalt Autoencoder for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2204.08227/) (2022)
- [The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models and its Applications](https://mlsanity.com/paper/2412.09726/) (2024)
- [On the Activation Function Dependence of the Spectral Bias of Neural Networks](https://mlsanity.com/paper/2208.04924/) (2022)
- [Frequency Domain-based Dataset Distillation](https://mlsanity.com/paper/2311.08819/) (2023)
- [Implicit regularization of dropout](https://mlsanity.com/paper/2207.05952/) (2022)
- [Explaining the physics of transfer learning a data-driven subgrid-scale closure to a different turbulent flow](https://mlsanity.com/paper/2206.03198/) (2022)
- [Empirical Phase Diagram for Three-layer Neural Networks with Infinite Width](https://mlsanity.com/paper/2205.12101/) (2022)
- [An overview of condensation phenomenon in deep learning](https://mlsanity.com/paper/2504.09484/) (2025)
- [Loss Spike in Training Neural Networks](https://mlsanity.com/paper/2305.12133/) (2023)
- [Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learning](https://mlsanity.com/paper/2410.16105/) (2024)
- [Partition Speeds Up Learning Implicit Neural Representations Based on Exponential-Increase Hypothesis](https://mlsanity.com/paper/2310.14184/) (2023)
- [Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild](https://mlsanity.com/paper/2503.10065/) (2025)
- [Multi-Grade Deep Learning](https://mlsanity.com/paper/2302.00150/) (2023)
- [Towards Building More Robust Models with Frequency Bias](https://mlsanity.com/paper/2307.09763/) (2023)
- [Anchor function: a type of benchmark functions for studying language models](https://mlsanity.com/paper/2401.08309/) (2024)
- [Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrum](https://mlsanity.com/paper/2307.14531/) (2023)
- [Spectral Co-Distillation for Personalized Federated Learning](https://mlsanity.com/paper/2401.17124/) (2024)
- [DenoDet: Attention as Deformable Multi-Subspace Feature Denoising for Target Detection in SAR Images](https://mlsanity.com/paper/2406.02833/) (2024)
- [A Mathematical Framework for Learning Probability Distributions](https://mlsanity.com/paper/2212.11481/) (2022)
- [ReFocus: Reinforcing Mid-Frequency and Key-Frequency Modeling for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2502.16890/) (2025)
- [Spectral Bias in Variational Quantum Machine Learning](https://mlsanity.com/paper/2506.22555/) (2025)
- [Hipandas: Hyperspectral Image Joint Denoising and Super-Resolution by Image Fusion with the Panchromatic Image](https://mlsanity.com/paper/2412.04201/) (2024)
- [Unsupervised Multi-Parameter Inverse Solving for Reducing Ring Artifacts in 3D X-Ray CBCT](https://mlsanity.com/paper/2412.05853/) (2024)
- [Rotary Masked Autoencoders are Versatile Learners](https://mlsanity.com/paper/2505.20535/) (2025)
- [PDEfuncta: Spectrally-Aware Neural Representation for PDE Solution Modeling](https://mlsanity.com/paper/2506.12790/) (2025)
- [Temporal Subsampling Diminishes Small Spatial Scales in Recurrent Neural Network Emulators of Geophysical Turbulence](https://mlsanity.com/paper/2305.00100/) (2023)
- [A rationale from frequency perspective for grokking in training neural network](https://mlsanity.com/paper/2405.17479/) (2024)
- [Spectrum-Informed Multistage Neural Networks: Multiscale Function Approximators of Machine Precision](https://mlsanity.com/paper/2407.17213/) (2024)
- [Learning Singularity-Encoded Green's Functions with Application to Iterative Methods](https://mlsanity.com/paper/2509.11580/) (2025)

## Abstract (arXiv)

Understanding deep learning is increasingly emergent as it penetrates more and more into industry and science. In recent years, a research line from Fourier analysis sheds lights on this magical "black box" by showing a Frequency Principle (F-Principle or spectral bias) of the training behavior of deep neural networks (DNNs) -- DNNs often fit functions from low to high frequency during the training. The F-Principle is first demonstrated by onedimensional synthetic data followed by the verification in high-dimensional real datasets. A series of works subsequently enhance the validity of the F-Principle. This low-frequency implicit bias reveals the strength of neural network in learning low-frequency functions as well as its deficiency in learning high-frequency functions. Such understanding inspires the design of DNN-based algorithms in practical problems, explains experimental phenomena emerging in various scenarios, and further advances the study of deep learning from the frequency perspective. Although incomplete, we provide an overview of F-Principle and propose some open problems for future research.