# Frequency Principle in Deep Learning Beyond Gradient-descent-based Training

arXiv: https://arxiv.org/abs/2101.00747 · HTML: https://mlsanity.com/paper/2101.00747/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 4 Jan 2021.

## Builds on (most cited first, 17 of 17)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks](https://mlsanity.com/paper/1901.06523/) (2019)
- [Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks](https://mlsanity.com/paper/2002.02561/) (2020)
- [Training behavior of deep neural network in frequency domain](https://mlsanity.com/paper/1807.01251/) (2018)
- [Towards Understanding the Spectral Bias of Deep Learning](https://mlsanity.com/paper/1912.01198/) (2019)
- [Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks](https://mlsanity.com/paper/1909.11957/) (2019)
- [The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies](https://mlsanity.com/paper/1906.00425/) (2019)
- [Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains](https://mlsanity.com/paper/2007.11207/) (2020)
- [A Fine-Grained Spectral Perspective on Neural Networks](https://mlsanity.com/paper/1907.10599/) (2019)
- [Machine Learning from a Continuous Viewpoint](https://mlsanity.com/paper/1912.12777/) (2019)
- [Theory of the Frequency Principle for General Deep Neural Networks](https://mlsanity.com/paper/1906.09235/) (2019)
- [Deep frequency principle towards understanding why deeper learning is faster](https://mlsanity.com/paper/2007.14313/) (2020)
- [Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks](https://mlsanity.com/paper/1905.10264/) (2019)
- [On the exact computation of linear frequency principle dynamics and its generalization](https://mlsanity.com/paper/2010.08153/) (2020)
- [The Slow Deterioration of the Generalization Error of the Random Feature Model](https://mlsanity.com/paper/2008.05621/) (2020)
- [Frequency-Aware Reconstruction of Fluid Simulations with Generative Networks](https://mlsanity.com/paper/1912.08776/) (2019)
- [Fourier-domain Variational Formulation and Its Well-posedness for Supervised Learning](https://mlsanity.com/paper/2012.03238/) (2020)

## Built on by (most cited first, 4 of 4)

- [Overview frequency principle/spectral bias in deep learning](https://mlsanity.com/paper/2201.07395/) (2022)
- [On the Activation Function Dependence of the Spectral Bias of Neural Networks](https://mlsanity.com/paper/2208.04924/) (2022)
- [Fast Fourier Correlation is a Highly Efficient and Accurate Feature Attribution Algorithm from the Perspective of Control Theory and Game Theory](https://mlsanity.com/paper/2504.02016/) (2025)
- [Comprehensive and Reliable Feature Attribution for Diverse Modalities and Models via Frequency-Domain Insights](https://mlsanity.com/paper/2411.18343/) (2024)

## Abstract (arXiv)

Frequency perspective recently makes progress in understanding deep learning. It has been widely verified in both empirical and theoretical studies that deep neural networks (DNNs) often fit the target function from low to high frequency, namely Frequency Principle (F-Principle). F-Principle sheds light on the strength and the weakness of DNNs and inspires a series of subsequent works, including theoretical studies, empirical studies and the design of efficient DNN structures etc. Previous works examine the F-Principle in gradient-descent-based training. It remains unclear whether gradient-descent-based training is a necessary condition for the F-Principle. In this paper, we show that the F-Principle exists stably in the training process of DNNs with non-gradient-descent-based training, including optimization algorithms with gradient information, such as conjugate gradient and BFGS, and algorithms without gradient information, such as Powell's method and Particle Swarm Optimization. These empirical studies show the universality of the F-Principle and provide hints for further study of F-Principle.