mlsanityData to 7 Oct 2026

On the Inductive Bias of Neural Tangent Kernels

arXiv 1905.12173 ↗First posted 29 May 2019stat.ML cs.LGcited by 177

Not on the map. Why?

Abstract (arXiv)

State-of-the-art neural networks are heavily over-parameterized, making the optimization algorithm a crucial ingredient for learning predictive models with good generalization properties. A recent line of work has shown that in a certain over-parameterized regime, the learning dynamics of gradient descent are governed by a certain kernel obtained at initialization, called the neural tangent kernel. We study the inductive bias of learning in such a regime by analyzing this kernel and the corresponding function space (RKHS). In particular, we study smoothness, approximation, and stability properties of functions with finite norm, including stability to image deformations in the case of convolutional networks, and compare to other known kernels for similar architectures.

Key facts. Cited by 177 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 35; 2022: 36; 2023: 29; 2024: 28; 2025: 31; 2026: 17.

177 papers.

Sort:

By area:

  1. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  2. 2023A Comprehensive Survey of Dataset Distillationcited by 114
  3. 2021Learning with invariances in random features and kernel modelscited by 89
  4. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  5. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  6. 2023NeuRBF: A Neural Fields Representation with Adaptive Radial Basis Functionscited by 65
  7. 2021Learning in High Dimension Always Amounts to Extrapolationcited by 62
  8. 2021SAPE: Spatially-Adaptive Progressive Encoding for Neural Optimizationcited by 59
  9. 2021What can linearized neural networks actually say about generalization?cited by 49
  10. 2022Graph Neural Networks are Inherently Good Generalizers: Insights by Bridging GNNs and MLPscited by 42
  11. 2023The SSL Interplay: Augmentations, Inductive Bias, and Generalizationcited by 40
  12. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 39
38 more
  1. 2021Rethinking Positional Encodingcited by 36
  2. 2024Improved Implicit Neural Representation with Fourier Reparameterized Trainingcited by 36
  3. 2021Scaling Neural Tangent Kernels via Sketching and Random Featurescited by 32
  4. 2021Approximation and Learning with Deep Convolutional Models: a Kernel Perspectivecited by 31
  5. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learningcited by 31
  6. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  7. 2022DINER: Disorder-Invariant Implicit Neural Representationcited by 30
  8. 2023On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domainscited by 30
  9. 2024Neural Networks Learn Statistics of Increasing Complexitycited by 30
  10. 2021Batch Normalization Orthogonalizes Representations in Deep Random Networkscited by 28
  11. 2023Generalization Ability of Wide Neural Networks on ℝcited by 28
  12. 2021Deep Neural Networks as Point Estimates for Deep Gaussian Processescited by 26
  13. 2021Locality defeats the curse of dimensionality in convolutional teacher-student scenarioscited by 26
  14. 2023Simplicity Bias in 1-Hidden Layer Neural Networkscited by 26
  15. 2023Disorder-invariant Implicit Neural Representationcited by 26
  16. 2021A Theory of Neural Tangent Kernel Alignment and Its Influence on Trainingcited by 25
  17. 2025Generalization through variance: how noise shapes inductive biases in diffusion modelscited by 23
  18. 2021Neural Active Learning with Performance Guaranteescited by 22
  19. 2022Neural Tangent Kernel: A Surveycited by 22
  20. 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
  21. 2021Out-of-Distribution Generalization in Kernel Regressioncited by 21
  22. 2021A Neural Tangent Kernel Perspective of GANscited by 21
  23. 2022What Can the Neural Tangent Kernel Tell Us About Adversarial Robustness?cited by 21
  24. 2021Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networkscited by 20
  25. 2022Single Model Uncertainty Estimation via Stochastic Data Centeringcited by 20
  26. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
  27. 2022The Spectral Bias of Polynomial Neural Networkscited by 19
  28. 2023On the Asymptotic Learning Curves of Kernel Ridge Regression under Power-law Decaycited by 19
  29. 2022Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?cited by 18
  30. 2022Overcoming the Spectral Bias of Neural Value Approximationcited by 18
  31. 2022Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization)cited by 18
  32. 2021Simple, Fast, and Flexible Framework for Matrix Completion with Infinite Width Neural Networkscited by 17
  33. 2021How Does Knowledge Graph Embedding Extrapolate to Unseen Data: A Semantic Evidence Viewcited by 17
  34. 2024U-Nets as Belief Propagation: Efficient Classification, Denoising, and Diffusion in Generative Hierarchical Modelscited by 17
  35. 2021Shift Invariance Can Reduce Adversarial Robustnesscited by 16
  36. 2022Trading Positional Complexity vs. Deepness in Coordinate Networkscited by 16
  37. 2022What Can Be Learnt With Wide Convolutional Neural Networks?cited by 16
  38. 2023Kernel interpolation generalizes poorlycited by 16

50 most cited of 177 All 177 (JSON)