mlsanityData to 7 Oct 2026

Frequency Bias in Neural Networks for Input of Non-Uniform Density

arXiv 2003.04560 ↗First posted 10 Mar 2020cs.LG stat.MLcited by 140

Not on the map. Why?

Abstract (arXiv)

Recent works have partly attributed the generalization ability of over-parameterized neural networks to frequency bias -- networks trained with gradient descent on data drawn from a uniform distribution find a low frequency fit before high frequency ones. As realistic training sets are not drawn from a uniform distribution, we here use the Neural Tangent Kernel (NTK) model to explore the effect of variable density on training dynamics. Our results, which combine analytic and empirical observations, show that when learning a pure harmonic function of frequency kappa, convergence at a point x ∈ Sphered-1 occurs in time O(kappad/p(x)) where p(x) denotes the local density at x. Specifically, for data in Sphere1 we analytically derive the eigenfunctions of the kernel associated with the NTK for two-layer networks. We further prove convergence results for deep, fully connected networks with respect to the spectral decomposition of the NTK. Our empirical study highlights similarities and differences between deep and shallow networks in this model.

Key facts. Cited by 140 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 28; 2023: 28; 2024: 30; 2025: 25; 2026: 18.

140 papers.

Sort:

By area:

  1. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  2. 2023FreeU: Free Lunch in Diffusion U-Netcited by 195
  3. 2021COTR: Correspondence Transformer for Matching Across Imagescited by 182
  4. 2021Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPscited by 134
  5. 2022Diffusion Probabilistic Model Made Slimcited by 121
  6. 2023An Expert's Guide to Training Physics-informed Neural Networkscited by 119
  7. 2022Overview frequency principle/spectral bias in deep learningcited by 68
  8. 2021Geometry-Consistent Neural Shape Representation with Implicit Displacement Fieldscited by 60
  9. 2024Kolmogorov-Arnold Transformercited by 56
  10. 2023Cascaded Local Implicit Transformer for Arbitrary-Scale Super-Resolutioncited by 51
  11. 2022Relaxing Equivariance Constraints with Non-stationary Continuous Filterscited by 41
  12. 2022Generalizable Implicit Neural Representations via Instance Pattern Composerscited by 38
38 more
  1. 2023Sphere2Vec: A General-Purpose Location Representation Learning over a Spherical Surface for Large-Scale Geospatial Predictionscited by 38
  2. 2021Spectral Bias in Practice: The Role of Function Frequency in Generalizationcited by 36
  3. 2024On the expressiveness and spectral bias of KANscited by 36
  4. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  5. 2025TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problemscited by 33
  6. 2022On Implicit Bias in Overparameterized Bilevel Optimizationcited by 30
  7. 2024Neural Networks Learn Statistics of Increasing Complexitycited by 30
  8. 2022Implicit Full Waveform Inversion with Deep Neural Representationcited by 26
  9. 2022On the Activation Function Dependence of the Spectral Bias of Neural Networkscited by 25
  10. 2024The Clever Hans Effect in Unsupervised Learningcited by 24
  11. 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
  12. 2024Improving Point-based Crowd Counting and Localization Based on Auxiliary Point Guidancecited by 22
  13. 2021A Neural Tangent Kernel Perspective of GANscited by 21
  14. 2022What Can the Neural Tangent Kernel Tell Us About Adversarial Robustness?cited by 21
  15. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  16. 2021Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networkscited by 20
  17. 2022Diffeomorphic Image Registration with Neural Velocity Fieldcited by 20
  18. 2022Intrinsic Neural Fields: Learning Functions on Manifoldscited by 20
  19. 2022Attention Beats Concatenation for Conditioning Neural Fieldscited by 20
  20. 2022PINs: Progressive Implicit Networks for Multi-Scale Neural Representationscited by 19
  21. 2022The Spectral Bias of Polynomial Neural Networkscited by 19
  22. 2023Interpretable Deep Clustering for Tabular Datacited by 19
  23. 2022Learning Deep Time-index Models for Time Series Forecastingcited by 18
  24. 2021Functional Regularization for Reinforcement Learning via Learned Fourier Featurescited by 17
  25. 2023Spectral-wise Implicit Neural Representation for Hyperspectral Image Reconstructioncited by 17
  26. 2021Shift Invariance Can Reduce Adversarial Robustnesscited by 16
  27. 2023Locality-Aware Generalizable Implicit Neural Representationcited by 16
  28. 2024Tuning Frequency Bias of State Space Modelscited by 16
  29. 2022On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernelscited by 15
  30. 2024Towards Understanding Inductive Bias in Transformers: A View From Infinitycited by 13
  31. 2024Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimensioncited by 13
  32. 2022Failure and success of the spectral bias prediction for Kernel Ridge Regression: the case of low-dimensional datacited by 12
  33. 2022A Kernel Perspective of Skip Connections in Convolutional Networkscited by 12
  34. 2023Sound propagation in realistic interactive 3D scenes with parameterized sources using deep neural operatorscited by 12
  35. 2022Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regimecited by 11
  36. 2022A view of mini-batch SGD via generating functions: conditions of convergence, phase transitions, benefit from negative momentacited by 11
  37. 2023Fourier Neural Operator Surrogate Model to Predict 3D Seismic Waves Propagationcited by 11
  38. 2024Simplicity Bias of Two-Layer Networks beyond Linearly Separable Datacited by 11

50 most cited of 140 All 140 (JSON)