mlsanityData to 7 Oct 2026

Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study

arXiv 2209.07736 ↗First posted 16 Sep 2022cs.LG cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Neural tangent kernel (NTK) is a powerful tool to analyze training dynamics of neural networks and their generalization bounds. The study on NTK has been devoted to typical neural network architectures, but it is incomplete for neural networks with Hadamard products (NNs-Hp), e.g., StyleGAN and polynomial neural networks (PNNs). In this work, we derive the finite-width NTK formulation for a special class of NNs-Hp, i.e., polynomial neural networks. We prove their equivalence to the kernel regression predictor with the associated NTK, which expands the application scope of NTK. Based on our results, we elucidate the separation of PNNs over standard neural networks with respect to extrapolation and spectral bias. Our two key insights are that when compared to standard neural networks, PNNs can fit more complicated functions in the extrapolation regime and admit a slower eigenvalue decay of the respective NTK, leading to a faster learning towards high-frequency functions. Besides, our theoretical results can be extended to other types of NNs-Hp, which expand the scope of our work. Our empirical results validate the separations in broader classes of NNs-Hp, which provide a good justification for a deeper understanding of neural architectures.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 44 of its 68 references that are in the map. First posted on arXiv 16 Sep 2022.

Builds on

44 of 68 references.

References · 44

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2017Non-local Neural Networkscited by 2,290
  6. 2019Invariant Risk Minimizationcited by 1,832
  7. 2018On the Spectral Bias of Neural Networkscited by 1,321
  8. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  9. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  10. 2018On Lazy Training in Differentiable Programmingcited by 671
  11. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  12. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
32 more
  1. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  2. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  3. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  4. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  5. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  6. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  7. 2020The Risks of Invariant Risk Minimizationcited by 262
  8. 2020How Neural Networks Extrapolate: From Feedforward to Graph Neural Networkscited by 218
  9. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  10. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 173
  11. 2019Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernelscited by 160
  12. 2019A review of domain adaptation without target labelscited by 154
  13. 2018Learning Equations for Extrapolation and Controlcited by 151
  14. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  15. 2016Extrapolation and learning equationscited by 106
  16. 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
  17. 2020The Recurrent Neural Tangent Kernelcited by 67
  18. 2020Deep Polynomial Neural Networkscited by 63
  19. 2021Learning in High Dimension Always Amounts to Extrapolationcited by 62
  20. 2020Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspectivecited by 60
  21. 2020Convolutional Tensor-Train LSTM for Spatio-temporal Learningcited by 54
  22. 2020Pi-nets: Deep Polynomial Neural Networkscited by 39
  23. 2020Learning Representations that Support Extrapolationcited by 36
  24. 2020Concentration Inequalities for Statistical Inferencecited by 24
  25. 2021Augmenting Deep Classifiers with Polynomial Neural Networkscited by 21
  26. 2021Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networkscited by 20
  27. 2020Kernel-Based Smoothness Analysis of Residual Networkscited by 19
  28. 2022The Spectral Bias of Polynomial Neural Networkscited by 19
  29. 2021CoPE: Conditional image generation using Polynomial Expansionscited by 12
  30. 2019PolyGAN: High-Order Polynomial Generatorscited by 10
  31. 2022Controlling the Complexity and Lipschitz Constant improves polynomial netscited by 5
  32. 2019Performing Arithmetic Using a Neural Network Trained on Digit Permutation Pairscited by 2

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 3; 2025: 1.

7 papers.

  1. 2023Regularization of polynomial networks for image recognitioncited by 13
  2. 2025Hadamard product in deep learning: Introduction, Advances and Challengescited by 9
  3. 2024Learnable Activation Functions in Physics-Informed Neural Networks for Solving Partial Differential Equationscited by 7
  4. 2024Comparing Spectral Bias and Robustness For Two-Layer Neural Networks: SGD vs Adaptive Random Fourier Featurescited by 3
  5. 2023Deep Neural Networks Tend To Extrapolate Predictablycited by 1
  6. 2023TMPNN: High-Order Polynomial Regression Based on Taylor Map Factorizationcited by 0
  7. 2024The Extrapolation Power of Implicit Modelscited by 0