mlsanityData to 7 Oct 2026

Theory of the Frequency Principle for General Deep Neural Networks

arXiv 1906.09235 ↗First posted 21 Jun 2019cs.LG math.OC stat.MLcited by 45

Not on the map. Why?

Abstract (arXiv)

Along with fruitful applications of Deep Neural Networks (DNNs) to realistic problems, recently, some empirical studies of DNNs reported a universal phenomenon of Frequency Principle (F-Principle): a DNN tends to learn a target function from low to high frequencies during the training. The F-Principle has been very useful in providing both qualitative and quantitative understandings of DNNs. In this paper, we rigorously investigate the F-Principle for the training dynamics of a general DNN at three stages: initial stage, intermediate stage, and final stage. For each stage, a theorem is provided in terms of proper quantities characterizing the F-Principle. Our results are general in the sense that they work for multilayer networks with general activation functions, population densities of data, and a large class of loss functions. Our work lays a theoretical foundation of the F-Principle for a better understanding of the training process of DNNs.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 6; 2022: 9; 2023: 8; 2024: 8; 2025: 7; 2026: 6.

45 papers.

Sort:

By area:

  1. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  2. 2022Overview frequency principle/spectral bias in deep learningcited by 68
  3. 2023Image Shortcut Squeezing: Countering Perturbative Availability Poisons with Compressioncited by 39
  4. 2022Implicit regularization of dropoutcited by 24
  5. 2021Embedding Principle: a hierarchical structure of loss landscape of deep neural networkscited by 21
  6. 2022On Numerical Integration in Neural Ordinary Differential Equationscited by 21
  7. 2021Reproducing Activation Function for Deep Learningcited by 19
  8. 2024No Time to Train: Empowering Non-Parametric Networks for Few-shot 3D Scene Segmentationcited by 18
  9. 2023Learning Gabor Texture Features for Fine-Grained Recognitioncited by 16
  10. 2023Loss Spike in Training Neural Networkscited by 15
  11. 2021The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimationcited by 14
  12. 2023Machine learning and domain decomposition methods -- a surveycited by 14
33 more
  1. 2023Regularize implicit neural representation by itselfcited by 13
  2. 2024Simplicity Bias of Two-Layer Networks beyond Linearly Separable Datacited by 11
  3. 2021Subspace Decomposition based DNN algorithm for elliptic type multi-scale PDEscited by 10
  4. 2022Defects of Convolutional Decoder Networks in Frequency Representationcited by 10
  5. 2024Anchor function: a type of benchmark functions for studying language modelscited by 10
  6. 2025FreqKV: Key-Value Compression in Frequency Domain for Context Window Extensioncited by 7
  7. 2023Spectral-DP: Differentially Private Deep Learning through Spectral Perturbation and Filteringcited by 6
  8. 2022Towards Inadequately Pre-trained Models in Transfer Learningcited by 5
  9. 2022How You Start Matters for Generalizationcited by 5
  10. 2022Invisible Backdoor Attacks Using Data Poisoning in the Frequency Domaincited by 5
  11. 2021Frequency Principle in Deep Learning Beyond Gradient-descent-based Trainingcited by 4
  12. 2024Understanding the dynamics of the frequency bias in neural networkscited by 4
  13. 2021An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Networkcited by 3
  14. 2022Wavelet Regularization Benefits Adversarial Trainingcited by 3
  15. 2024A rationale from frequency perspective for grokking in training neural networkcited by 3
  16. 2025Spatiotemporal wall pressure forecast of a rectangular cylinder with physics-aware DeepU-Fourier neural networkcited by 2
  17. 2022Data-driven soliton mappings for integrable fractional nonlinear wave equations via deep learning with Fourier neural operatorcited by 1
  18. 2024GlobalMamba: Global Image Serialization for Vision Mambacited by 1
  19. 2024Ensemble learning of the atrial fiber orientation with physics-informed neural networkscited by 1
  20. 2025FW-GAN: Frequency-Driven Handwriting Synthesis with Wave-Modulated MLP Generatorcited by 1
  21. 2025FreIE: Low-Frequency Spectral Bias in Neural Networks for Time-Series Taskscited by 1
  22. 2025SHAP values through General Fourier Representations: Theory and Applicationscited by 1
  23. 2026Multigrade Neural Network Approximationcited by 1
  24. 2026Layer-wise Geometric Approximation Rates for Deep Networkscited by 1
  25. 2023Phase-shifted Adversarial Trainingcited by 0
  26. 2023Enhanced physics-informed neural networks with domain scaling and residual correction methods for multi-frequency elliptic problemscited by 0
  27. 2024Loss Jump During Loss Switch in Solving PDEs with Neural Networkscited by 0
  28. 2025Is the Frequency Principle always valid?cited by 0
  29. 2025A Greedy PDE Router for Blending Neural Operators and Classical Methodscited by 0
  30. 2026Improving Variational Autoencoder using Random Fourier Transformation: An Aviation Safety Anomaly Detection Case-Studycited by 0
  31. 2026Pseudo-differential-enhanced physics-informed neural networkscited by 0
  32. 2026Shearlet Neural Operators for Anisotropic-Shock-Dominated and Multi-scale parametric partial differential equationscited by 0
  33. 2026SpiS-GAN: Spiral-Modulated Handwriting Synthesis with Star Operationcited by 0