# The Spectral Bias of Polynomial Neural Networks

arXiv: https://arxiv.org/abs/2202.13473 · HTML: https://mlsanity.com/paper/2202.13473/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 48 references that are in the map.
- First posted on arXiv 27 Feb 2022.

## Builds on (most cited first, 30 of 33)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks](https://mlsanity.com/paper/1901.08584/) (2019)
- [Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks](https://mlsanity.com/paper/1901.06523/) (2019)
- [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014)
- [Highway Networks](https://mlsanity.com/paper/1505.00387/) (2015)
- [Implicit Bias of Gradient Descent on Linear Convolutional Networks](https://mlsanity.com/paper/1806.00468/) (2018)
- [Deep learning generalizes because the parameter-function map is biased towards simple functions](https://mlsanity.com/paper/1805.08522/) (2018)
- [On the Inductive Bias of Neural Tangent Kernels](https://mlsanity.com/paper/1905.12173/) (2019)
- [Towards Understanding the Spectral Bias of Deep Learning](https://mlsanity.com/paper/1912.01198/) (2019)
- [Frequency Bias in Neural Networks for Input of Non-Uniform Density](https://mlsanity.com/paper/2003.04560/) (2020)
- [The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies](https://mlsanity.com/paper/1906.00425/) (2019)
- [On the Power of Over-parametrization in Neural Networks with Quadratic Activation](https://mlsanity.com/paper/1803.01206/) (2018)
- [A Fine-Grained Spectral Perspective on Neural Networks](https://mlsanity.com/paper/1907.10599/) (2019)
- [On the Expressive Power of Deep Polynomial Neural Networks](https://mlsanity.com/paper/1905.12207/) (2019)
- [Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions](https://mlsanity.com/paper/2006.15459/) (2020)
- [Pi-nets: Deep Polynomial Neural Networks](https://mlsanity.com/paper/2003.03828/) (2020)
- [Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime](https://mlsanity.com/paper/2006.12297/) (2020)

## Built on by (most cited first, 19 of 19)

- [Neural Redshift: Random Networks are not Random Functions](https://mlsanity.com/paper/2403.02241/) (2024)
- [Polynomial Neural Fields for Subband Decomposition and Manipulation](https://mlsanity.com/paper/2302.04862/) (2023)
- [Regularization of polynomial networks for image recognition](https://mlsanity.com/paper/2303.13896/) (2023)
- [Identifiability of Deep Polynomial Neural Networks](https://mlsanity.com/paper/2506.17093/) (2025)
- [Fast Training of Sinusoidal Neural Fields via Scaling Initialization](https://mlsanity.com/paper/2410.04779/) (2024)
- [Hadamard product in deep learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2504.13112/) (2025)
- [Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study](https://mlsanity.com/paper/2209.07736/) (2022)
- [Learnable Activation Functions in Physics-Informed Neural Networks for Solving Partial Differential Equations](https://mlsanity.com/paper/2411.15111/) (2024)
- [Robust NAS under adversarial training: benchmark, theory, and beyond](https://mlsanity.com/paper/2403.13134/) (2024)
- [Deep ReLU Networks Have Surprisingly Simple Polytopes](https://mlsanity.com/paper/2305.09145/) (2023)
- [Sound and Complete Verification of Polynomial Networks](https://mlsanity.com/paper/2209.07235/) (2022)
- [HOIN: High-Order Implicit Neural Representations](https://mlsanity.com/paper/2404.14674/) (2024)
- [PNeRV: A Polynomial Neural Representation for Videos](https://mlsanity.com/paper/2406.19299/) (2024)
- [On the Study of Sample Complexity for Polynomial Neural Networks](https://mlsanity.com/paper/2207.08896/) (2022)
- [Spectral Regularization Allows Data-frugal Learning over Combinatorial Spaces](https://mlsanity.com/paper/2210.02604/) (2022)
- [Graph Contrastive Learning with Low-Rank Regularization and Low-Rank Attention for Noisy Node Classification](https://mlsanity.com/paper/2402.09600/) (2024)
- [Is the Frequency Principle always valid?](https://mlsanity.com/paper/2508.17323/) (2025)
- [Shallow Neural Networks Learn Low-Degree Spherical Polynomials with Feature Learning by Learnable Channel Attention](https://mlsanity.com/paper/2512.20562/) (2025)
- [Gradient Descent with Projection Finds Over-Parameterized Neural Networks for Learning Low-Degree Polynomials with Nearly Minimax Optimal Rate](https://mlsanity.com/paper/2603.21062/) (2026)

## Abstract (arXiv)

Polynomial neural networks (PNNs) have been recently shown to be particularly effective at image generation and face recognition, where high-frequency information is critical. Previous studies have revealed that neural networks demonstrate a textitspectral bias towards low-frequency functions, which yields faster learning of low-frequency components during training. Inspired by such studies, we conduct a spectral analysis of the Neural Tangent Kernel (NTK) of PNNs. We find that the Pi-Net family, i.e., a recently proposed parametrization of PNNs, speeds up the learning of the higher frequencies. We verify the theoretical bias through extensive experiments. We expect our analysis to provide novel insights into designing architectures and learning frameworks by incorporating multiplicative interactions via polynomials.