# Frequency Bias in Neural Networks for Input of Non-Uniform Density

arXiv: https://arxiv.org/abs/2003.04560 · HTML: https://mlsanity.com/paper/2003.04560/

## Key facts

- Cited by 140 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Mar 2020.

## Built on by (most cited first, 30 of 140)

- [Hyena Hierarchy: Towards Larger Convolutional Language Models](https://mlsanity.com/paper/2302.10866/) (2023)
- [FreeU: Free Lunch in Diffusion U-Net](https://mlsanity.com/paper/2309.11497/) (2023)
- [COTR: Correspondence Transformer for Matching Across Images](https://mlsanity.com/paper/2103.14167/) (2021)
- [Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPs](https://mlsanity.com/paper/2111.15135/) (2021)
- [An Expert's Guide to Training Physics-informed Neural Networks](https://mlsanity.com/paper/2308.08468/) (2023)
- [Diffusion Probabilistic Model Made Slim](https://mlsanity.com/paper/2211.17106/) (2022)
- [Overview frequency principle/spectral bias in deep learning](https://mlsanity.com/paper/2201.07395/) (2022)
- [Geometry-Consistent Neural Shape Representation with Implicit Displacement Fields](https://mlsanity.com/paper/2106.05187/) (2021)
- [Kolmogorov-Arnold Transformer](https://mlsanity.com/paper/2409.10594/) (2024)
- [Cascaded Local Implicit Transformer for Arbitrary-Scale Super-Resolution](https://mlsanity.com/paper/2303.16513/) (2023)
- [Relaxing Equivariance Constraints with Non-stationary Continuous Filters](https://mlsanity.com/paper/2204.07178/) (2022)
- [Generalizable Implicit Neural Representations via Instance Pattern Composers](https://mlsanity.com/paper/2211.13223/) (2022)
- [Sphere2Vec: A General-Purpose Location Representation Learning over a Spherical Surface for Large-Scale Geospatial Predictions](https://mlsanity.com/paper/2306.17624/) (2023)
- [Spectral Bias in Practice: The Role of Function Frequency in Generalization](https://mlsanity.com/paper/2110.02424/) (2021)
- [On the expressiveness and spectral bias of KANs](https://mlsanity.com/paper/2410.01803/) (2024)
- [TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems](https://mlsanity.com/paper/2502.02527/) (2025)
- [Representation Learning for Tabular Data: A Comprehensive Survey](https://mlsanity.com/paper/2504.16109/) (2025)
- [Neural Networks Learn Statistics of Increasing Complexity](https://mlsanity.com/paper/2402.04362/) (2024)
- [On Implicit Bias in Overparameterized Bilevel Optimization](https://mlsanity.com/paper/2212.14032/) (2022)
- [There Will Be a Scientific Theory of Deep Learning](https://mlsanity.com/paper/2604.21691/) (2026)
- [On the Activation Function Dependence of the Spectral Bias of Neural Networks](https://mlsanity.com/paper/2208.04924/) (2022)
- [Implicit Full Waveform Inversion with Deep Neural Representation](https://mlsanity.com/paper/2209.03525/) (2022)
- [The Clever Hans Effect in Unsupervised Learning](https://mlsanity.com/paper/2408.08041/) (2024)
- [Characterizing the Spectrum of the NTK via a Power Series Expansion](https://mlsanity.com/paper/2211.07844/) (2022)
- [Improving Point-based Crowd Counting and Localization Based on Auxiliary Point Guidance](https://mlsanity.com/paper/2405.10589/) (2024)
- [Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networks](https://mlsanity.com/paper/2104.03093/) (2021)
- [A Neural Tangent Kernel Perspective of GANs](https://mlsanity.com/paper/2106.05566/) (2021)
- [What Can the Neural Tangent Kernel Tell Us About Adversarial Robustness?](https://mlsanity.com/paper/2210.05577/) (2022)
- [Diffeomorphic Image Registration with Neural Velocity Field](https://mlsanity.com/paper/2202.12498/) (2022)
- [Intrinsic Neural Fields: Learning Functions on Manifolds](https://mlsanity.com/paper/2203.07967/) (2022)

## Abstract (arXiv)

Recent works have partly attributed the generalization ability of over-parameterized neural networks to frequency bias -- networks trained with gradient descent on data drawn from a uniform distribution find a low frequency fit before high frequency ones. As realistic training sets are not drawn from a uniform distribution, we here use the Neural Tangent Kernel (NTK) model to explore the effect of variable density on training dynamics. Our results, which combine analytic and empirical observations, show that when learning a pure harmonic function of frequency kappa, convergence at a point x ∈ Sphered-1 occurs in time O(kappad/p(x)) where p(x) denotes the local density at x. Specifically, for data in Sphere1 we analytically derive the eigenfunctions of the kernel associated with the NTK for two-layer networks. We further prove convergence results for deep, fully connected networks with respect to the spectral decomposition of the NTK. Our empirical study highlights similarities and differences between deep and shallow networks in this model.