mlsanityData to 7 Oct 2026

Overview frequency principle/spectral bias in deep learning

arXiv 2201.07395 ↗First posted 19 Jan 2022cs.LGcited by 68

Not on the map. Why?

Abstract (arXiv)

Understanding deep learning is increasingly emergent as it penetrates more and more into industry and science. In recent years, a research line from Fourier analysis sheds lights on this magical "black box" by showing a Frequency Principle (F-Principle or spectral bias) of the training behavior of deep neural networks (DNNs) -- DNNs often fit functions from low to high frequency during the training. The F-Principle is first demonstrated by onedimensional synthetic data followed by the verification in high-dimensional real datasets. A series of works subsequently enhance the validity of the F-Principle. This low-frequency implicit bias reveals the strength of neural network in learning low-frequency functions as well as its deficiency in learning high-frequency functions. Such understanding inspires the design of DNN-based algorithms in practical problems, explains experimental phenomena emerging in various scenarios, and further advances the study of deep learning from the frequency perspective. Although incomplete, we provide an overview of F-Principle and propose some open problems for future research.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 90 of its 135 references that are in the map. First posted on arXiv 19 Jan 2022.

Builds on

90 of 135 references.

References · 90

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  7. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  8. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  9. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  10. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  11. 2017A Closer Look at Memorization in Deep Networkscited by 936
  12. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
38 more
  1. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  2. 2020Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humanscited by 634
  3. 2019DeepXDE: A deep learning library for solving differential equationscited by 617
  4. 2017The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problemscited by 581
  5. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  6. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  7. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  8. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  9. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  10. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 360
  11. 2020Neural Additive Models: Interpretable Machine Learning with Neural Netscited by 318
  12. 2020On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networkscited by 279
  13. 2014The Loss Surfaces of Multilayer Networkscited by 278
  14. 2020Learned Initializations for Optimizing Coordinate-Based Neural Representationscited by 240
  15. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  16. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  17. 2020Neural Reflectance Fields for Appearance Acquisitioncited by 196
  18. 2017Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equationscited by 188
  19. 2018Training behavior of deep neural network in frequency domaincited by 174
  20. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 173
  21. 2019Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networkscited by 160
  22. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 158
  23. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  24. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
  25. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  26. 2020Object-Centric Neural Scene Renderingcited by 90
  27. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 84
  28. 2017Failures of Gradient-Based Deep Learningcited by 79
  29. 2020Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domainscited by 78
  30. 2020Algorithms for Solving High Dimensional PDEs: From Nonlinear Monte Carlo to Machine Learningcited by 77
  31. 2018A Priori Estimates of the Population Risk for Two-layer Neural Networkscited by 76
  32. 2020Positional Encoding as Spatial Inductive Bias in GANscited by 69
  33. 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
  34. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 65
  35. 2021On the Frequency Bias of Generative Modelscited by 60
  36. 2018The committee machine: Computational to statistical gaps in learning a two-layers neural networkcited by 58
  37. 2020The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networkscited by 58
  38. 2019Machine Learning from a Continuous Viewpointcited by 50

50 most cited of 90 All 90 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 7; 2024: 14; 2025: 23; 2026: 16.

68 papers.

Sort:

By area:

  1. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 218
  2. 2022The Devil is in the Frequency: Geminated Gestalt Autoencoder for Self-Supervised Visual Pre-Trainingcited by 33
  3. 2024The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models and its Applicationscited by 32
  4. 2023Frequency Domain-based Dataset Distillationcited by 28
  5. 2022On the Activation Function Dependence of the Spectral Bias of Neural Networkscited by 25
  6. 2022Implicit regularization of dropoutcited by 24
  7. 2022Explaining the physics of transfer learning a data-driven subgrid-scale closure to a different turbulent flowcited by 20
  8. 2022Empirical Phase Diagram for Three-layer Neural Networks with Infinite Widthcited by 17
  9. 2023Loss Spike in Training Neural Networkscited by 15
  10. 2024Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learningcited by 14
  11. 2023Partition Speeds Up Learning Implicit Neural Representations Based on Exponential-Increase Hypothesiscited by 13
  12. 2025Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wildcited by 13
38 more
  1. 2025An overview of condensation phenomenon in deep learningcited by 13
  2. 2023Multi-Grade Deep Learningcited by 12
  3. 2023Towards Building More Robust Models with Frequency Biascited by 12
  4. 2024Anchor function: a type of benchmark functions for studying language modelscited by 10
  5. 2023Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrumcited by 9
  6. 2024Spectral Co-Distillation for Personalized Federated Learningcited by 8
  7. 2024DenoDet: Attention as Deformable Multi-Subspace Feature Denoising for Target Detection in SAR Imagescited by 8
  8. 2022A Mathematical Framework for Learning Probability Distributionscited by 7
  9. 2025ReFocus: Reinforcing Mid-Frequency and Key-Frequency Modeling for Multivariate Time Series Forecastingcited by 5
  10. 2025Spectral Bias in Variational Quantum Machine Learningcited by 5
  11. 2025Rotary Masked Autoencoders are Versatile Learnerscited by 4
  12. 2025PDEfuncta: Spectrally-Aware Neural Representation for PDE Solution Modelingcited by 4
  13. 2023Temporal Subsampling Diminishes Small Spatial Scales in Recurrent Neural Network Emulators of Geophysical Turbulencecited by 3
  14. 2024A rationale from frequency perspective for grokking in training neural networkcited by 3
  15. 2024Hipandas: Hyperspectral Image Joint Denoising and Super-Resolution by Image Fusion with the Panchromatic Imagecited by 3
  16. 2024Unsupervised Multi-Parameter Inverse Solving for Reducing Ring Artifacts in 3D X-Ray CBCTcited by 3
  17. 2025Learning Singularity-Encoded Green's Functions with Application to Iterative Methodscited by 3
  18. 2025Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic datacited by 3
  19. 2022QFF: Quantized Fourier Features for Neural Field Representationscited by 2
  20. 2024Spectrum-Informed Multistage Neural Networks: Multiscale Function Approximators of Machine Precisioncited by 2
  21. 2024TAUDiff: Highly efficient kilometer-scale downscaling using generative diffusion modelscited by 2
  22. 2025Learn Singularly Perturbed Solutions via Homotopy Dynamicscited by 2
  23. 2025Hypernym Bias: Unraveling Deep Classifier Training Dynamics through the Lens of Class Hierarchycited by 2
  24. 2025Initial Model Incorporation for Deep Learning FWI: Pretraining or Denormalization?cited by 2
  25. 2025Probability Signature: Bridging Data Semantics and Embedding Structure in Language Modelscited by 2
  26. 2024GlobalMamba: Global Image Serialization for Vision Mambacited by 1
  27. 2024ε-rank and the Staircase Phenomenon: New Insights into Neural Network Training Dynamicscited by 1
  28. 2025Toward a Low-Cost Perception System in Autonomous Vehicles: A Spectrum Learning Approachcited by 1
  29. 2025Data-driven quantum Koopman method for simulating nonlinear dynamicscited by 1
  30. 2025A physics-aware deep learning model for shear band formation around collapsing pores in shocked reactive materialscited by 1
  31. 2025SONAR: Spectral-Contrastive Audio Residuals for Generalizable Deepfake Detectioncited by 1
  32. 2025Multi-resolution Physics-Aware Recurrent Convolutional Neural Network for Complex Flowscited by 1
  33. 2026Position: Certifiable State Integrity Should Be Built from Local Validity, Not Global Scalecited by 1
  34. 2026Safe Fairness Guarantees Without Demographics in Classification: Spectral Uncertainty Set Perspectivecited by 1
  35. 2026Unveiling the Mechanism of Continuous Representation Full-Waveform Inversion: A Wave Based Neural Tangent Kernel Frameworkcited by 1
  36. 2026Detail Consistent Stage-Wise Distillation for Efficient 3D MRI Segmentationcited by 1
  37. 2022Limitation of Characterizing Implicit Regularization by Data-independent Functionscited by 0
  38. 2024Architectural Strategies for the optimization of Physics-Informed Neural Networkscited by 0

50 most cited of 68 All 68 (JSON)