# On the asymptotics of wide networks with polynomial activations

arXiv: https://arxiv.org/abs/2006.06687 · HTML: https://mlsanity.com/paper/2006.06687/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2020.

## Built on by (most cited first, 18 of 18)

- [Neural Networks as Kernel Learners: The Silent Alignment Effect](https://mlsanity.com/paper/2111.00034/) (2021)
- [Scaling and renormalization in high-dimensional regression](https://mlsanity.com/paper/2405.00592/) (2024)
- [Dataset Distillation with Convexified Implicit Gradients](https://mlsanity.com/paper/2302.06755/) (2023)
- [A Theory of Neural Tangent Kernel Alignment and Its Influence on Training](https://mlsanity.com/paper/2105.14301/) (2021)
- [Contrasting random and learned features in deep Bayesian linear regression](https://mlsanity.com/paper/2203.00573/) (2022)
- [Non-asymptotic approximations of neural networks by Gaussian processes](https://mlsanity.com/paper/2102.08668/) (2021)
- [Meta-Principled Family of Hyperparameter Scaling Strategies](https://mlsanity.com/paper/2210.04909/) (2022)
- [Evolution of Neural Tangent Kernels under Benign and Adversarial Training](https://mlsanity.com/paper/2210.12030/) (2022)
- [Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillation](https://mlsanity.com/paper/2302.01428/) (2023)
- [Rate of Convergence of Polynomial Networks to Gaussian Processes](https://mlsanity.com/paper/2111.03175/) (2021)
- [Structures of Neural Network Effective Theories](https://mlsanity.com/paper/2305.02334/) (2023)
- [Non-asymptotic approximations of Gaussian neural networks via second-order Poincar'e inequalities](https://mlsanity.com/paper/2304.04010/) (2023)
- [From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning](https://mlsanity.com/paper/2502.03210/) (2025)
- [Deep Stable neural networks: large-width asymptotics and convergence rates](https://mlsanity.com/paper/2108.02316/) (2021)
- [Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applications](https://mlsanity.com/paper/2111.12143/) (2021)
- [Exact marginal prior distributions of finite Bayesian neural networks](https://mlsanity.com/paper/2104.11734/) (2021)
- [Gradient Flow Through Diagram Expansions: Learning Regimes and Explicit Solutions](https://mlsanity.com/paper/2602.04548/) (2026)
- [K"ahler landscapes for complex neural network descents and guarantees including a search and destroy of the Calabi-Yau manifold](https://mlsanity.com/paper/2608.19584/) (2026)

## Abstract (arXiv)

We consider an existing conjecture addressing the asymptotic behavior of neural networks in the large width limit. The results that follow from this conjecture include tight bounds on the behavior of wide networks during stochastic gradient descent, and a derivation of their finite-width dynamics. We prove the conjecture for deep networks with polynomial activation functions, greatly extending the validity of these results. Finally, we point out a difference in the asymptotic behavior of networks with analytic (and non-linear) activation functions and those with piecewise-linear activations such as ReLU.