# Training behavior of deep neural network in frequency domain

arXiv: https://arxiv.org/abs/1807.01251 · HTML: https://mlsanity.com/paper/1807.01251/

## Key facts

- Cited by 174 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2018.

## Built on by (most cited first, 30 of 174)

- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [FreeU: Free Lunch in Diffusion U-Net](https://mlsanity.com/paper/2309.11497/) (2023)
- [Omnigrok: Grokking Beyond Algorithmic Data](https://mlsanity.com/paper/2210.01117/) (2022)
- [Diffusion Probabilistic Model Made Slim](https://mlsanity.com/paper/2211.17106/) (2022)
- [GARF: Gaussian Activated Radiance Fields for High Fidelity Reconstruction and Pose Estimation](https://mlsanity.com/paper/2204.05735/) (2022)
- [Overview frequency principle/spectral bias in deep learning](https://mlsanity.com/paper/2201.07395/) (2022)
- [HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging](https://mlsanity.com/paper/2203.02149/) (2022)
- [Learning in Sinusoidal Spaces with Physics-Informed Neural Networks](https://mlsanity.com/paper/2109.09338/) (2021)
- [Geometry-Consistent Neural Shape Representation with Implicit Displacement Fields](https://mlsanity.com/paper/2106.05187/) (2021)
- [SAPE: Spatially-Adaptive Progressive Encoding for Neural Optimization](https://mlsanity.com/paper/2104.09125/) (2021)
- [One-Shot Transfer Learning of Physics-Informed Neural Networks](https://mlsanity.com/paper/2110.11286/) (2021)
- [Open-Vocabulary Segmentation with Semantic-Assisted Calibration](https://mlsanity.com/paper/2312.04089/) (2023)
- [ADD: Frequency Attention and Multi-View based Knowledge Distillation to Detect Low-Quality Compressed Deepfake Images](https://mlsanity.com/paper/2112.03553/) (2021)
- [Spectrum-guided Multi-granularity Referring Video Object Segmentation](https://mlsanity.com/paper/2307.13537/) (2023)
- [Adaptive Frequency Filters As Efficient Global Token Mixers](https://mlsanity.com/paper/2307.14008/) (2023)
- [Inverse-Dirichlet Weighting Enables Reliable Training of Physics Informed Neural Networks](https://mlsanity.com/paper/2107.00940/) (2021)
- [Deep Frequency Filtering for Domain Generalization](https://mlsanity.com/paper/2203.12198/) (2022)
- [Neural Redshift: Random Networks are not Random Functions](https://mlsanity.com/paper/2403.02241/) (2024)
- [Fighting deepfakes by detecting GAN DCT anomalies](https://mlsanity.com/paper/2101.09781/) (2021)
- [Image Shortcut Squeezing: Countering Perturbative Availability Poisons with Compression](https://mlsanity.com/paper/2301.13838/) (2023)
- [Embedding Principle of Loss Landscape of Deep Neural Networks](https://mlsanity.com/paper/2105.14573/) (2021)
- [Spectral Bias in Practice: The Role of Function Frequency in Generalization](https://mlsanity.com/paper/2110.02424/) (2021)
- [On the expressiveness and spectral bias of KANs](https://mlsanity.com/paper/2410.01803/) (2024)
- [Towards Understanding the Condensation of Neural Networks at Initial Training](https://mlsanity.com/paper/2105.11686/) (2021)
- [Neural Networks Learn Statistics of Increasing Complexity](https://mlsanity.com/paper/2402.04362/) (2024)
- [Sensitivity as a Complexity Measure for Sequence Classification Tasks](https://mlsanity.com/paper/2104.10343/) (2021)
- [FreqFed: A Frequency Analysis-Based Approach for Mitigating Poisoning Attacks in Federated Learning](https://mlsanity.com/paper/2312.04432/) (2023)
- [The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networks](https://mlsanity.com/paper/2110.03922/) (2021)
- [Personalized Retrogress-Resilient Framework for Real-World Medical Federated Learning](https://mlsanity.com/paper/2110.00394/) (2021)
- [Implicit regularization of dropout](https://mlsanity.com/paper/2207.05952/) (2022)

## Abstract (arXiv)

Why deep neural networks (DNNs) capable of overfitting often generalize well in practice is a mystery [#zhang2016understanding]. To find a potential mechanism, we focus on the study of implicit biases underlying the training process of DNNs. In this work, for both real and synthetic datasets, we empirically find that a DNN with common settings first quickly captures the dominant low-frequency components, and then relatively slowly captures the high-frequency ones. We call this phenomenon Frequency Principle (F-Principle). The F-Principle can be observed over DNNs of various structures, activation functions, and training algorithms in our experiments. We also illustrate how the F-Principle help understand the effect of early-stopping as well as the generalization of DNNs. This F-Principle potentially provides insights into a general principle underlying DNN optimization and generalization.