mlsanityData to 14 Oct 2026

On the Spectral Bias of Neural Networks

arXiv 1806.08734 ↗First posted 22 Jun 2018stat.ML cs.LGcited by 1,341

Not on the map. Why?

Cited by 81 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Neural networks are known to be a class of highly expressive functions able to fit even random input-output mappings with 100% accuracy. In this work, we present properties of neural networks that complement this aspect of expressivity. By using tools from Fourier analysis, we show that deep ReLU networks are biased towards low frequency functions, meaning that they cannot have local fluctuations without affecting their global behavior. Intuitively, this property is in line with the observation that over-parameterized networks find simple patterns that generalize across data samples. We also investigate how the shape of the data manifold affects expressivity by showing evidence that learning high frequencies gets easier with increasing manifold complexity, and present a theoretical understanding of this behavior. Finally, we study the robustness of the frequency components with respect to parameter perturbation, to develop the intuition that the parameters must be finely tuned to express high frequency functions.

Key facts. Cited by 1,341 papers in the map (data through 14 Oct 2026). First posted on arXiv 22 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 84; 2022: 149; 2023: 183; 2024: 239; 2025: 320; 2026: 365.

1,341 papers.

Sort:

By area:

  1. 2021Neural Fields in Visual Computing and Beyondcited by 540
  2. 2021Deep Neural Networks and Tabular Data: A Surveycited by 491
  3. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 381
  4. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 334
  5. 2021NeRV: Neural Representations for Videoscited by 256
  6. 2023Implicit Diffusion Models for Continuous Super-Resolutioncited by 218
  7. 2023Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Modelscited by 203
  8. 2022Why do tree-based models still outperform deep learning on tabular data?cited by 199
  9. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
  10. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 186
  11. 2022PointAvatar: Deformable Point-based Head Avatars from Videoscited by 184
  12. 2023Co-SLAM: Joint Coordinate and Sparse Parametric Encodings for Neural Real-Time SLAMcited by 184
38 more
  1. 2021COTR: Correspondence Transformer for Matching Across Imagescited by 183
  2. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  3. 20213D Infomax improves GNNs for Molecular Property Predictioncited by 146
  4. 2021Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equationscited by 145
  5. 2021Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPscited by 135
  6. 2021MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesiscited by 133
  7. 2021Local Texture Estimator for Implicit Representation Functioncited by 127
  8. 2023An Expert's Guide to Training Physics-informed Neural Networkscited by 126
  9. 2022NAF: Neural Attenuation Fields for Sparse-View CBCT Reconstructioncited by 108
  10. 2022Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imagingcited by 107
  11. 2024A comprehensive and FAIR comparison between MLP and KAN representations for differential equations and operator networkscited by 104
  12. 2021imGHUM: Implicit Generative Models of 3D Human Shape and Articulated Posecited by 103
  13. 2021A Structured Dictionary Perspective on Implicit Neural Representationscited by 100
  14. 2023HNeRV: A Hybrid Neural Representation for Videoscited by 100
  15. 2021SurfEmb: Dense and Continuous Correspondence Distributions for Object Pose Estimation with Learnt Surface Embeddingscited by 95
  16. 2023Implicit Neural Representation for Cooperative Low-light Image Enhancementcited by 92
  17. 2023LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generationcited by 92
  18. 2023Implicit Neural Representation in Medical Imaging: A Comparative Surveycited by 90
  19. 2021Long-time integration of parametric evolution equations with physics-informed DeepONetscited by 89
  20. 2021Improved architectures and training algorithms for deep operator networkscited by 86
  21. 2022GARF: Gaussian Activated Radiance Fields for High Fidelity Reconstruction and Pose Estimationcited by 86
  22. 2023In-Context Learning through the Bayesian Prismcited by 83
  23. 2023Temporal Dynamic Quantization for Diffusion Modelscited by 81
  24. 2024Robust agents learn causal world modelscited by 81
  25. 2025Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Trainingcited by 81
  26. 2023FINER: Flexible spectral-bias tuning in Implicit NEural Representation by Variable-periodic Activation Functionscited by 78
  27. 2024Why are Sensitive Functions Hard for Transformers?cited by 74
  28. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  29. 2024From PINNs to PIKANs: Recent Advances in Physics-Informed Machine Learningcited by 73
  30. 2024PirateNets: Physics-informed Deep Learning with Residual Adaptive Networkscited by 72
  31. 2021PointMixer: MLP-Mixer for Point Cloud Understandingcited by 71
  32. 2022Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functionscited by 70
  33. 2022Stitch it in Time: GAN-Based Facial Editing of Real Videoscited by 68
  34. 2023Can Physics-Informed Neural Networks beat the Finite Element Method?cited by 68
  35. 2022HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imagingcited by 66
  36. 2023On the Foundations of Shortcut Learningcited by 65
  37. 2021Learning in Sinusoidal Spaces with Physics-Informed Neural Networkscited by 62
  38. 2022Wavelet Score-Based Generative Modelingcited by 62

50 most cited of 1,341 All 1,341 (JSON)