# Is the Frequency Principle always valid?

arXiv: https://arxiv.org/abs/2508.17323 · HTML: https://mlsanity.com/paper/2508.17323/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 24 Aug 2025.

## Builds on (most cited first, 16 of 16)

- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks](https://mlsanity.com/paper/1901.06523/) (2019)
- [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014)
- [Training behavior of deep neural network in frequency domain](https://mlsanity.com/paper/1807.01251/) (2018)
- [Natural Language Processing: State of The Art, Current Trends and Challenges](https://mlsanity.com/paper/1708.05148/) (2017)
- [Deep Learning vs. Traditional Computer Vision](https://mlsanity.com/paper/1910.13796/) (2019)
- [Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains](https://mlsanity.com/paper/2007.11207/) (2020)
- [Overview frequency principle/spectral bias in deep learning](https://mlsanity.com/paper/2201.07395/) (2022)
- [Machine Learning from a Continuous Viewpoint](https://mlsanity.com/paper/1912.12777/) (2019)
- [A Survey of Deep Learning Techniques for Neural Machine Translation](https://mlsanity.com/paper/2002.07526/) (2020)
- [Theory of the Frequency Principle for General Deep Neural Networks](https://mlsanity.com/paper/1906.09235/) (2019)
- [On the Activation Function Dependence of the Spectral Bias of Neural Networks](https://mlsanity.com/paper/2208.04924/) (2022)
- [Multi-scale Deep Neural Networks for Solving High Dimensional PDEs](https://mlsanity.com/paper/1910.11710/) (2019)
- [The Spectral Bias of Polynomial Neural Networks](https://mlsanity.com/paper/2202.13473/) (2022)
- [Plateau Phenomenon in Gradient Descent Training of ReLU networks: Explanation, Quantification and Avoidance](https://mlsanity.com/paper/2007.07213/) (2020)

## Abstract (arXiv)

We investigate the learning dynamics of shallow ReLU neural networks on the unit sphere (S2subsetℝ3) in polar coordinates ((τ,φ)), considering both fixed and trainable neuron directions (wi). For fixed weights, spherical harmonic expansions reveal an intrinsic low-frequency preference with coefficients decaying as (O(ℓ5/2/2ℓ)), typically leading to the Frequency Principle (FP) of lower-frequency-first learning. However, this principle can be violated under specific initial conditions or error distributions. With trainable weights, an additional rotation term in the harmonic evolution equations preserves exponential decay with decay order (O(ℓ7/2/2ℓ)) factor, also leading to the FP of lower-frequency-first learning. But like fixed weights case, the principle can be violated under specific initial conditions or error distributions. Our numerical results demonstrate that trainable directions increase learning complexity and can either maintain a low-frequency advantage or enable faster high-frequency emergence. This analysis suggests the FP should be viewed as a tendency rather than a rule on curved domains like (S2), providing insights into how direction updates and harmonic expansions shape frequency-dependent learning.