Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theories
Not on the map. Why?
Abstract (arXiv)
A primary advantage of neural networks lies in their feature learning characteristics, which is challenging to theoretically analyze due to the complexity of their training dynamics. We examine feature learning and its potential benefits for generalization from a statistical perspective. After reviewing the neural tangent kernel (NTK) theory and recent results in kernel regression, which address the generalization issue of sufficiently wide neural networks, we examine limitations and implications of the fixed kernel theory (as the NTK theory) and review recent theoretical advancements in feature learning. Moving beyond theories with fixed features, we consider neural networks as adaptive feature models. Finally, we propose an over-parameterized Gaussian sequence model as a prototype for the adaptive feature model to study feature learning characteristics and motivate their future analysis for neural networks.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 94 of its 131 references that are in the map. First posted on arXiv 25 Dec 2024.
Builds on
94 of 131 references.
References · 94
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2019Similarity of Neural Network Representations Revisitedcited by 1,695
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
- 2018On Lazy Training in Differentiable Programmingcited by 671
- 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
- 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
- 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
- 2019Benign Overfitting in Linear Regressioncited by 544
- 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
- 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
- 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
- 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
38 more
- 2021Deep learning: a statistical viewpointcited by 239
- 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
- 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
- 2018To understand deep learning we need to understand kernel learningcited by 186
- 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
- 2020Feature Learning in Infinite-Width Neural Networkscited by 182
- 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
- 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
- 2019Linearized two-layers neural networks in high dimensioncited by 155
- 2020Benign overfitting in ridge regressioncited by 152
- 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
- 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
- 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
- 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 141
- 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
- 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
- 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
- 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
- 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
- 2017Sobolev Norm Learning Rates for Regularized Least-Squares Algorithmcited by 113
- 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
- 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
- 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
- 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
- 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
- 2021Learning with invariances in random features and kernel modelscited by 89
- 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
- 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
- 2019Implicit Regularization for Optimal Sparse Recoverycited by 78
- 2019Non-Gaussian processes and neural networks at finite widthscited by 78
- 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
- 2018Statistical Optimality of Stochastic Gradient Descent on Hard Learning Problems through Multiple Passescited by 69
- 2016Optimal Rates For Regularization Of Statistical Inverse Learning Problemscited by 68
- 2019Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobiancited by 67
- 2020Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspectivecited by 60
- 2018Optimal Rates for Spectral Algorithms with Least-Squares Regression over Hilbert Spacescited by 59
- 2018Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularizationcited by 52
- 2021What can linearized neural networks actually say about generalization?cited by 49
50 most cited of 94 All 94 (JSON)
Built on by
Not cited yet.