On the asymptotics of wide networks with polynomial activations
Not on the map. Why?
Abstract (arXiv)
We consider an existing conjecture addressing the asymptotic behavior of neural networks in the large width limit. The results that follow from this conjecture include tight bounds on the behavior of wide networks during stochastic gradient descent, and a derivation of their finite-width dynamics. We prove the conjecture for deep networks with polynomial activation functions, greatly extending the validity of these results. Finally, we point out a difference in the asymptotic behavior of networks with analytic (and non-linear) activation functions and those with piecewise-linear activations such as ReLU.
Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Jun 2020.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 7; 2022: 3; 2023: 4; 2024: 1; 2025: 1; 2026: 1.
17 papers.
Sort:
- 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
- 2023Dataset Distillation with Convexified Implicit Gradientscited by 46
- 2024Scaling and renormalization in high-dimensional regressioncited by 45
- 2021A Theory of Neural Tangent Kernel Alignment and Its Influence on Trainingcited by 25
- 2022Contrasting random and learned features in deep Bayesian linear regressioncited by 24
- 2021Non-asymptotic approximations of neural networks by Gaussian processescited by 18
- 2022Meta-Principled Family of Hyperparameter Scaling Strategiescited by 18
- 2022Evolution of Neural Tangent Kernels under Benign and Adversarial Trainingcited by 14
- 2023Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillationcited by 14
- 2023Structures of Neural Network Effective Theoriescited by 13
- 2021Rate of Convergence of Polynomial Networks to Gaussian Processescited by 12
- 2023Non-asymptotic approximations of Gaussian neural networks via second-order Poincar'e inequalitiescited by 10
5 more
- 2025From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learningcited by 10
- 2021Deep Stable neural networks: large-width asymptotics and convergence ratescited by 8
- 2021Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applicationscited by 8
- 2021Exact marginal prior distributions of finite Bayesian neural networkscited by 1
- 2026Gradient Flow Through Diagram Expansions: Learning Regimes and Explicit Solutionscited by 1
- 2026Gradient Flow Through Diagram Expansions: Learning Regimes and Explicit Solutions4 Feb 2026
- 2025From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning5 Feb 2025
- 2024Scaling and renormalization in high-dimensional regression1 May 2024
- 2023Structures of Neural Network Effective Theories3 May 2023
- 2023Non-asymptotic approximations of Gaussian neural networks via second-order Poincar'e inequalities8 Apr 2023
- 2023Dataset Distillation with Convexified Implicit Gradients13 Feb 2023
- 2023Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillation2 Feb 2023
- 2022Evolution of Neural Tangent Kernels under Benign and Adversarial Training21 Oct 2022
- 2022Meta-Principled Family of Hyperparameter Scaling Strategies10 Oct 2022
- 2022Contrasting random and learned features in deep Bayesian linear regression1 Mar 2022
- 2021Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applications23 Nov 2021
- 2021Rate of Convergence of Polynomial Networks to Gaussian Processes4 Nov 2021
5 more
- 2021Neural Networks as Kernel Learners: The Silent Alignment Effect29 Oct 2021
- 2021Deep Stable neural networks: large-width asymptotics and convergence rates2 Aug 2021
- 2021A Theory of Neural Tangent Kernel Alignment and Its Influence on Training29 May 2021
- 2021Exact marginal prior distributions of finite Bayesian neural networks23 Apr 2021
- 2021Non-asymptotic approximations of neural networks by Gaussian processes17 Feb 2021
- 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
- 2023Dataset Distillation with Convexified Implicit Gradientscited by 46
- 2024Scaling and renormalization in high-dimensional regressioncited by 45
- 2021A Theory of Neural Tangent Kernel Alignment and Its Influence on Trainingcited by 25
- 2022Contrasting random and learned features in deep Bayesian linear regressioncited by 24
- 2021Non-asymptotic approximations of neural networks by Gaussian processescited by 18
- 2022Meta-Principled Family of Hyperparameter Scaling Strategiescited by 18
- 2022Evolution of Neural Tangent Kernels under Benign and Adversarial Trainingcited by 14
- 2023Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillationcited by 14
- 2023Structures of Neural Network Effective Theoriescited by 13
- 2021Rate of Convergence of Polynomial Networks to Gaussian Processescited by 12
- 2023Non-asymptotic approximations of Gaussian neural networks via second-order Poincar'e inequalitiescited by 10
12 most cited of 17 All 17 (JSON)
- 2026Gradient Flow Through Diagram Expansions: Learning Regimes and Explicit Solutions4 Feb 2026
- 2025From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning5 Feb 2025
- 2024Scaling and renormalization in high-dimensional regression1 May 2024
- 2023Structures of Neural Network Effective Theories3 May 2023
- 2023Non-asymptotic approximations of Gaussian neural networks via second-order Poincar'e inequalities8 Apr 2023
- 2023Dataset Distillation with Convexified Implicit Gradients13 Feb 2023
- 2023Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillation2 Feb 2023
- 2022Evolution of Neural Tangent Kernels under Benign and Adversarial Training21 Oct 2022
- 2022Meta-Principled Family of Hyperparameter Scaling Strategies10 Oct 2022
- 2022Contrasting random and learned features in deep Bayesian linear regression1 Mar 2022
- 2021Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applications23 Nov 2021
- 2021Rate of Convergence of Polynomial Networks to Gaussian Processes4 Nov 2021
12 newest of 17 All 17 (JSON)