mlsanityData to 7 Oct 2026

Understanding the dynamics of the frequency bias in neural networks

arXiv 2405.14957 ↗First posted 23 May 2024cs.LG cs.AI stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

Recent works have shown that traditional Neural Network (NN) architectures display a marked frequency bias in the learning process. Namely, the NN first learns the low-frequency features before learning the high-frequency ones. In this study, we rigorously develop a partial differential equation (PDE) that unravels the frequency dynamics of the error for a 2-layer NN in the Neural Tangent Kernel regime. Furthermore, using this insight, we explicitly demonstrate how an appropriate choice of distributions for the initialization weights can eliminate or control the frequency bias. We focus our study on the Fourier Features model, an NN where the first layer has sine and cosine activation functions, with frequencies sampled from a prescribed distribution. In this setup, we experimentally validate our theoretical results and compare the NN dynamics to the solution of the PDE using the finite element method. Finally, we empirically show that the same principle extends to multi-layer NNs.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 38 of its 50 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

38 of 50 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  4. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  5. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  6. 2018On the Spectral Bias of Neural Networkscited by 1,321
  7. 2017A Closer Look at Memorization in Deep Networkscited by 936
  8. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  9. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  10. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  11. 2021Neural Fields in Visual Computing and Beyondcited by 539
  12. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 360
26 more
  1. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
  2. 2020On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networkscited by 279
  3. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  4. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  5. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  6. 2018Training behavior of deep neural network in frequency domaincited by 174
  7. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 173
  8. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
  9. 2021Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPscited by 134
  10. 2023Implicit Neural Representation in Medical Imaging: A Comparative Surveycited by 88
  11. 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
  12. 2021Seeing Implicit Neural Representations as Fourier Seriescited by 50
  13. 2019Theory of the Frequency Principle for General Deep Neural Networkscited by 45
  14. 2023Identifying Spurious Biases Early in Training through the Lens of Simplicity Biascited by 44
  15. 2018Understanding training and generalization in deep learning by Fourier analysiscited by 43
  16. 2021Rethinking Positional Encodingcited by 36
  17. 2020Deep frequency principle towards understanding why deeper learning is fastercited by 30
  18. 2023Simplicity Bias in 1-Hidden Layer Neural Networkscited by 26
  19. 2022On the Activation Function Dependence of the Spectral Bias of Neural Networkscited by 25
  20. 2020On the exact computation of linear frequency principle dynamics and its generalizationcited by 16
  21. 2022On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernelscited by 15
  22. 2021Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networkscited by 14
  23. 2023Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrumcited by 9
  24. 2023On progressive sharpening, flat minima and generalisationcited by 6
  25. 2021An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Networkcited by 3
  26. 2016Tight Hardness Results for Distance and Centrality Problems in Constant Degree Graphscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2025Distributional Biases in Post-Training: A Markovian Analysis of Reasoning Trajectoriescited by 3
  2. 2024Ensemble learning of the atrial fiber orientation with physics-informed neural networkscited by 1
  3. 2025Beyond Universal Approximation Theorems: Algorithmic Uniform Approximation by Neural Networks Trained with Noisy Datacited by 1
  4. 2026Pseudo-differential-enhanced physics-informed neural networkscited by 0