mlsanityData to 7 Oct 2026

The Spectral Bias of Polynomial Neural Networks

arXiv 2202.13473 ↗First posted 27 Feb 2022cs.LG cs.CVcited by 19

Not on the map. Why?

Abstract (arXiv)

Polynomial neural networks (PNNs) have been recently shown to be particularly effective at image generation and face recognition, where high-frequency information is critical. Previous studies have revealed that neural networks demonstrate a textitspectral bias towards low-frequency functions, which yields faster learning of low-frequency components during training. Inspired by such studies, we conduct a spectral analysis of the Neural Tangent Kernel (NTK) of PNNs. We find that the Pi-Net family, i.e., a recently proposed parametrization of PNNs, speeds up the learning of the higher frequencies. We verify the theoretical bias through extensive experiments. We expect our analysis to provide novel insights into designing architectures and learning frameworks by incorporating multiplicative interactions via polynomials.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 33 of its 48 references that are in the map. First posted on arXiv 27 Feb 2022.

Builds on

33 of 48 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  6. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  7. 2018On the Spectral Bias of Neural Networkscited by 1,321
  8. 2017Deep Image Priorcited by 1,097
  9. 2017A Closer Look at Memorization in Deep Networkscited by 936
  10. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  11. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  12. 2018On Lazy Training in Differentiable Programmingcited by 671
21 more
  1. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  2. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  3. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  4. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 360
  5. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  6. 2015Highway Networkscited by 257
  7. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  8. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 180
  9. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  10. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 173
  11. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  12. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
  13. 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
  14. 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
  15. 2019On the Expressive Power of Deep Polynomial Neural Networkscited by 65
  16. 2020Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functionscited by 50
  17. 2020Pi-nets: Deep Polynomial Neural Networkscited by 39
  18. 2020Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regimecited by 35
  19. 2021Early-stopped neural networks are consistentcited by 34
  20. 2017SORT: Second-Order Response Transform for Visual Recognitioncited by 8
  21. 2021Poly-NL: Linear Complexity Non-local Layers with Polynomialscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 3; 2024: 7; 2025: 4; 2026: 1.

19 papers.

Sort:

  1. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  2. 2023Polynomial Neural Fields for Subband Decomposition and Manipulationcited by 22
  3. 2023Regularization of polynomial networks for image recognitioncited by 13
  4. 2025Identifiability of Deep Polynomial Neural Networkscited by 12
  5. 2024Fast Training of Sinusoidal Neural Fields via Scaling Initializationcited by 11
  6. 2025Hadamard product in deep learning: Introduction, Advances and Challengescited by 9
  7. 2022Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Studycited by 7
  8. 2024Learnable Activation Functions in Physics-Informed Neural Networks for Solving Partial Differential Equationscited by 7
  9. 2024Robust NAS under adversarial training: benchmark, theory, and beyondcited by 6
  10. 2023Deep ReLU Networks Have Surprisingly Simple Polytopescited by 5
  11. 2022Sound and Complete Verification of Polynomial Networkscited by 4
  12. 2024HOIN: High-Order Implicit Neural Representationscited by 3
7 more
  1. 2024PNeRV: A Polynomial Neural Representation for Videoscited by 2
  2. 2022On the Study of Sample Complexity for Polynomial Neural Networkscited by 1
  3. 2022Spectral Regularization Allows Data-frugal Learning over Combinatorial Spacescited by 1
  4. 2024Graph Contrastive Learning with Low-Rank Regularization and Low-Rank Attention for Noisy Node Classificationcited by 1
  5. 2025Is the Frequency Principle always valid?cited by 0
  6. 2025Shallow Neural Networks Learn Low-Degree Spherical Polynomials with Feature Learning by Learnable Channel Attentioncited by 0
  7. 2026Gradient Descent with Projection Finds Over-Parameterized Neural Networks for Learning Low-Degree Polynomials with Nearly Minimax Optimal Ratecited by 0