mlsanityData to 7 Oct 2026

Training behavior of deep neural network in frequency domain

arXiv 1807.01251 ↗First posted 3 Jul 2018cs.LG cs.AI cs.ITcited by 174

Not on the map. Why?

Abstract (arXiv)

Why deep neural networks (DNNs) capable of overfitting often generalize well in practice is a mystery [#zhang2016understanding]. To find a potential mechanism, we focus on the study of implicit biases underlying the training process of DNNs. In this work, for both real and synthetic datasets, we empirically find that a DNN with common settings first quickly captures the dominant low-frequency components, and then relatively slowly captures the high-frequency ones. We call this phenomenon Frequency Principle (F-Principle). The F-Principle can be observed over DNNs of various structures, activation functions, and training algorithms in our experiments. We also illustrate how the F-Principle help understand the effect of early-stopping as well as the generalization of DNNs. This F-Principle potentially provides insights into a general principle underlying DNN optimization and generalization.

Key facts. Cited by 174 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Jul 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 33; 2022: 32; 2023: 33; 2024: 27; 2025: 29; 2026: 20.

174 papers.

Sort:

By area:

  1. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  2. 2023FreeU: Free Lunch in Diffusion U-Netcited by 195
  3. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 142
  4. 2022Diffusion Probabilistic Model Made Slimcited by 121
  5. 2022GARF: Gaussian Activated Radiance Fields for High Fidelity Reconstruction and Pose Estimationcited by 86
  6. 2022Overview frequency principle/spectral bias in deep learningcited by 68
  7. 2022HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imagingcited by 66
  8. 2021Learning in Sinusoidal Spaces with Physics-Informed Neural Networkscited by 61
  9. 2021Geometry-Consistent Neural Shape Representation with Implicit Displacement Fieldscited by 60
  10. 2021SAPE: Spatially-Adaptive Progressive Encoding for Neural Optimizationcited by 59
  11. 2021One-Shot Transfer Learning of Physics-Informed Neural Networkscited by 49
  12. 2023Open-Vocabulary Segmentation with Semantic-Assisted Calibrationcited by 49
38 more
  1. 2021ADD: Frequency Attention and Multi-View based Knowledge Distillation to Detect Low-Quality Compressed Deepfake Imagescited by 48
  2. 2023Spectrum-guided Multi-granularity Referring Video Object Segmentationcited by 48
  3. 2023Adaptive Frequency Filters As Efficient Global Token Mixerscited by 45
  4. 2021Inverse-Dirichlet Weighting Enables Reliable Training of Physics Informed Neural Networkscited by 43
  5. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  6. 2022Deep Frequency Filtering for Domain Generalizationcited by 41
  7. 2021Fighting deepfakes by detecting GAN DCT anomaliescited by 39
  8. 2023Image Shortcut Squeezing: Countering Perturbative Availability Poisons with Compressioncited by 39
  9. 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 36
  10. 2021Spectral Bias in Practice: The Role of Function Frequency in Generalizationcited by 36
  11. 2024On the expressiveness and spectral bias of KANscited by 36
  12. 2021Towards Understanding the Condensation of Neural Networks at Initial Trainingcited by 31
  13. 2024Neural Networks Learn Statistics of Increasing Complexitycited by 30
  14. 2021Sensitivity as a Complexity Measure for Sequence Classification Taskscited by 28
  15. 2021Personalized Retrogress-Resilient Framework for Real-World Medical Federated Learningcited by 25
  16. 2021The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networkscited by 25
  17. 2023FreqFed: A Frequency Analysis-Based Approach for Mitigating Poisoning Attacks in Federated Learningcited by 25
  18. 2021MOD-Net: A Machine Learning Approach via Model-Operator-Data Network for Solving PDEscited by 24
  19. 2022Implicit regularization of dropoutcited by 24
  20. 2022Out-of-distribution Detection via Frequency-regularized Generative Modelscited by 23
  21. 2022Global Spectral Filter Memory Network for Video Object Segmentationcited by 22
  22. 2023I-MedSAM: Implicit Medical Image Segmentation with Segment Anythingcited by 22
  23. 2021Spline Positional Encoding for Learning 3D Implicit Signed Distance Fieldscited by 21
  24. 2021Embedding Principle: a hierarchical structure of loss landscape of deep neural networkscited by 21
  25. 2022On Numerical Integration in Neural Ordinary Differential Equationscited by 21
  26. 2024SpeGCL: Self-supervised Graph Spectrum Contrastive Learning without Positive Samplescited by 20
  27. 2024Beta Sampling is All You Need: Efficient Image Generation Strategy for Diffusion Models using Stepwise Spectral Analysiscited by 19
  28. 2025Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformerscited by 19
  29. 2021Backdoor Attack through Frequency Domaincited by 18
  30. 2021Multiple shooting for training neural differential equations on time seriescited by 17
  31. 2022Empirical Phase Diagram for Three-layer Neural Networks with Infinite Widthcited by 17
  32. 2022Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measurescited by 17
  33. 2023Explicit Visual Prompting for Universal Foreground Segmentationscited by 17
  34. 2023Learning Gabor Texture Features for Fine-Grained Recognitioncited by 16
  35. 2023Loss Spike in Training Neural Networkscited by 15
  36. 2021The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimationcited by 14
  37. 2021Solving Partial Differential Equations with Point Source Based on Physics-Informed Neural Networkscited by 14
  38. 2024Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learningcited by 14

50 most cited of 174 All 174 (JSON)