mlsanityData to 14 Oct 2026

On the Activation Function Dependence of the Spectral Bias of Neural Networks

arXiv 2208.04924 ↗First posted 9 Aug 2022cs.LGcited by 28

Not on the map. Why?

Abstract (arXiv)

Neural networks are universal function approximators which are known to generalize well despite being dramatically overparameterized. We study this phenomenon from the point of view of the spectral bias of neural networks. Our contributions are two-fold. First, we provide a theoretical explanation for the spectral bias of ReLU neural networks by leveraging connections with the theory of finite element methods. Second, based upon this theory we predict that switching the activation function to a piecewise linear B-spline, namely the Hat function, will remove this spectral bias, which we verify empirically in a variety of settings. Our empirical studies also show that neural networks with the Hat activation function are trained significantly faster using stochastic gradient descent and ADAM. Combined with previous work showing that the Hat activation function also improves generalization accuracy on image classification tasks, this indicates that using the Hat activation provides significant advantages over the ReLU on certain problems.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 26 of its 43 references that are in the map. First posted on arXiv 9 Aug 2022.

Builds on

26 of 43 references.

References · 26

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,238
  4. 2018On the Spectral Bias of Neural Networkscited by 1,341
  5. 2017A Closer Look at Memorization in Deep Networkscited by 942
  6. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  7. 2017Solving high-dimensional partial differential equations using deep learningcited by 556
  8. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 368
  9. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 174
  10. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  11. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 159
  12. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
14 more
  1. 2022Overview frequency principle/spectral bias in deep learningcited by 69
  2. 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
  3. 2020The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networkscited by 58
  4. 2018Understanding training and generalization in deep learning by Fourier analysiscited by 44
  5. 2021Spectral Bias in Practice: The Role of Function Frequency in Generalizationcited by 37
  6. 2019A Phase Shift Deep Neural Network for High Frequency Approximation and Wave Problemscited by 35
  7. 2020Spatial Frequency Bias in Convolutional Generative Adversarial Networkscited by 35
  8. 2019Pathological spectra of the Fisher information metric and its variants in deep neural networkscited by 32
  9. 2019Multi-scale Deep Neural Networks for Solving High Dimensional PDEscited by 25
  10. 2020On the exact computation of linear frequency principle dynamics and its generalizationcited by 16
  11. 2020Rethink the Connections among Generalization, Memorization and the Spectral Bias of DNNscited by 11
  12. 2019Frequency-Aware Reconstruction of Fluid Simulations with Generative Networkscited by 7
  13. 2019Neural Spectrum Alignment: Empirical Studycited by 6
  14. 2021Frequency Principle in Deep Learning Beyond Gradient-descent-based Trainingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 9; 2025: 11; 2026: 6.

28 papers.

Sort:

  1. 2022Reliable extrapolation of deep neural operators informed by physics or sparse observationscited by 62
  2. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  3. 2024On the expressiveness and spectral bias of KANscited by 37
  4. 2024Parametric Encoding with Attention and Convolution Mitigate Spectral Bias of Neural Partial Differential Equation Solverscited by 9
  5. 2024Learnable Activation Functions in Physics-Informed Neural Networks for Solving Partial Differential Equationscited by 7
  6. 2025High precision PINNs in unbounded domains: application to singularity formulation in PDEscited by 7
  7. 2023Homotopy Relaxation Training Algorithms for Infinite-Width Two-Layer ReLU Neural Networkscited by 4
  8. 2024Understanding the dynamics of the frequency bias in neural networkscited by 4
  9. 2025Leveraging KANs for Expedient Training of Multichannel MLPs via Preconditioning and Geometric Refinementcited by 4
  10. 2024Comparing Spectral Bias and Robustness For Two-Layer Neural Networks: SGD vs Adaptive Random Fourier Featurescited by 3
  11. 2024A Structure-Guided Gauss-Newton Method for Shallow ReLU Neural Networkcited by 3
  12. 2025Orthogonal greedy algorithm for linear operator learning with shallow neural networkcited by 3
16 more
  1. 2025Separated-Variable Spectral Neural Networks: A Physics-Informed Learning Approach for High-Frequency PDEscited by 3
  2. 2025Condition Numbers and Eigenvalue Spectra of Shallow Networks on Spherescited by 2
  3. 2024Deep Learning without Global Optimization by Random Fourier Neural Networkscited by 1
  4. 2025INR-Bench: A Unified Benchmark for Implicit Neural Representations in Multi-Domain Regression and Reconstructioncited by 1
  5. 2025Domain decomposition architectures and Gauss-Newton training for physics-informed neural networkscited by 1
  6. 2026On the Convergence Behavior of Preconditioned Gradient Descent Toward the Rich Learning Regimecited by 1
  7. 2026Multilevel Training for Kolmogorov Arnold Networkscited by 1
  8. 2024Efficient Shallow Ritz Method For 1D Diffusion-Reaction Problemscited by 0
  9. 2025An optimal Petrov-Galerkin framework for operator networkscited by 0
  10. 2025Is the Frequency Principle always valid?cited by 0
  11. 2025AR-KAN: Autoregressive-Weight-Enhanced Kolmogorov-Arnold Network for Time Series Forecastingcited by 0
  12. 2025Domain-Decomposed Graph Neural Network Surrogate Modeling for Ice Sheetscited by 0
  13. 2026Singularity Formation: Synergy in Theoretical, Numerical and Machine Learning Approachescited by 0
  14. 2026Autocorrelation Reintroduces Spectral Bias in KANs for Time Series Forecastingcited by 0
  15. 2026RepNN: Tackling spectral bias in deep neural networks for regression and PDE problems via parameter reparameterizationcited by 0
  16. 2026Sharp Approximation Rates for Neural Networks with Affine Latent Parameterizationscited by 0