A Special Case of Quadratic Extrapolation Under the Neural Tangent Kernel
Not on the map. Why?
Abstract (arXiv)
It has been demonstrated both theoretically and empirically that the ReLU MLP tends to extrapolate linearly for an out-of-distribution evaluation point. The machine learning literature provides ample analysis with respect to the mechanisms to which linearity is induced. However, the analysis of extrapolation at the origin under the NTK regime remains a more unexplored special case. In particular, the infinite-dimensional feature map induced by the neural tangent kernel is not translationally invariant. This means that the study of an out-of-distribution evaluation point very far from the origin is not equivalent to the evaluation of a point very near the origin. And since the feature map is rotation invariant, these two special cases may represent the most canonically extreme bounds of ReLU NTK extrapolation. Ultimately, it is this loose recognition of the two special cases of extrapolation that motivate the discovery of quadratic extrapolation for an evaluation close to the origin.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 13 of its 20 references that are in the map. First posted on arXiv 11 Dec 2025.
Builds on
13 of 20 references.
References · 13
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
- 2020How Neural Networks Extrapolate: From Feedforward to Graph Neural Networkscited by 218
- 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
- 2020Towards Understanding Hierarchical Learning: Benefits of Neural Representationscited by 46
- 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
- 2020On the infinite width limit of neural networks with a standard parameterizationcited by 38
- 2023On the Asymptotic Learning Curves of Kernel Ridge Regression under Power-law Decaycited by 19
- 2025Asymptotic Optimism of Random-Design Linear and Kernel Regression Modelscited by 2
- 2025Just One Layer Norm Guarantees Stable Extrapolationcited by 2
- 2025Why Cannot Neural Networks Master Extrapolation? Insights from Physical Lawscited by 2
- 2025Depth-induced NTK: Bridging Over-parameterized Neural Networks and Deep Neural Kernelscited by 1
1 more
Built on by
Not cited yet.