Weighted Neural Tangent Kernel: A Generalized and Improved Network-Induced Kernel
Not on the map. Why?
Abstract (arXiv)
The Neural Tangent Kernel (NTK) has recently attracted intense study, as it describes the evolution of an over-parameterized Neural Network (NN) trained by gradient descent. However, it is now well-known that gradient descent is not always a good optimizer for NNs, which can partially explain the unsatisfactory practical performance of the NTK regression estimator. In this paper, we introduce the Weighted Neural Tangent Kernel (WNTK), a generalized and improved tool, which can capture an over-parameterized NN's training dynamics under different optimizers. Theoretically, in the infinite-width limit, we prove: i) the stability of the WNTK at initialization and during training, and ii) the equivalence between the WNTK regression estimator and the corresponding NN estimator with different learning rates on different parameters. With the proposed weight update algorithm, both empirical and analytical WNTKs outperform the corresponding NTKs in numerical experiments.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 37 of its 55 references that are in the map. First posted on arXiv 22 Mar 2021.
Builds on
37 of 55 references.
References · 37
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
- 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
- 2018On Lazy Training in Differentiable Programmingcited by 671
- 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
- 2017Deep Neural Networks as Gaussian Processescited by 617
- 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
- 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
- 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
- 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
- 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
- 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
25 more
- 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
- 2018To understand deep learning we need to understand kernel learningcited by 186
- 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
- 2019Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernelscited by 160
- 2019Linearized two-layers neural networks in high dimensioncited by 155
- 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
- 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
- 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
- 2020Momentum Improves Normalized SGDcited by 136
- 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
- 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
- 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
- 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
- 2019Harnessing the Power of Infinitely Wide Deep Nets on Small-data Taskscited by 87
- 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
- 2020The Recurrent Neural Tangent Kernelcited by 67
- 2020Neural Kernels Without Tangentscited by 56
- 2017FreezeOut: Accelerate Training by Progressively Freezing Layerscited by 54
- 2020On the infinite width limit of neural networks with a standard parameterizationcited by 38
- 2015Steps Toward Deep Kernel Methods from Infinite Neural Networkscited by 28
- 2020Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticitycited by 18
- 2015Deep Convolutional Networks are Hierarchical Kernel Machinescited by 9
- 2018Learning Data-adaptive Nonparametric Kernelscited by 7
- 2019Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guaranteecited by 3
- 2020On the Empirical Neural Tangent Kernel of Standard Finite-Width Convolutional Neural Network Architecturescited by 3
Built on by
Citing papers per year: 2023: 1.
1 paper.