mlsanityData to 7 Oct 2026

Fast Finite Width Neural Tangent Kernel

arXiv 2206.08720 ↗First posted 17 Jun 2022cs.LG cs.AI stat.MLcited by 62

Not on the map. Why?

Abstract (arXiv)

The Neural Tangent Kernel (NTK), defined as Thetaθf(x1, x2) = left[partial f(θ, x1)big/partial θright] left[partial f(θ, x2)big/partial θright]T where left[partial f(θ, ·)big/partial θright] is a neural network (NN) Jacobian, has emerged as a central object of study in deep learning. In the infinite width limit, the NTK can sometimes be computed analytically and is useful for understanding training and generalization of NN architectures. At finite widths, the NTK is also used to better initialize NNs, compare the conditioning across models, perform architecture search, and do meta-learning. Unfortunately, the finite width NTK is notoriously expensive to compute, which severely limits its practical utility. We perform the first in-depth analysis of the compute and memory requirements for NTK computation in finite width networks. Leveraging the structure of neural networks, we further propose two novel algorithms that change the exponent of the compute and memory requirements of the finite width NTK, dramatically improving efficiency. Our algorithms can be applied in a black box fashion to any differentiable function, including those implementing neural networks. We open-source our implementations within the Neural Tangents package (arXiv:1912.02803) at https://github.com/google/neural-tangents.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 52 of its 76 references that are in the map. First posted on arXiv 17 Jun 2022.

Builds on

52 of 76 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2016Wide Residual Networkscited by 3,057
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  6. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  7. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  8. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  9. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  10. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  11. 2017Deep Neural Networks as Gaussian Processescited by 617
  12. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
38 more
  1. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  2. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  3. 2018TADAM: Task dependent adaptive metric for improved few-shot learningcited by 408
  4. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  5. 2021Explaining Neural Scaling Lawscited by 353
  6. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  7. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  8. 2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230
  9. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 215
  10. 2016Deep Information Propagationcited by 194
  11. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
  12. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  13. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  14. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
  15. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  16. 2019Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernelscited by 160
  17. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  18. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  19. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  20. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 139
  21. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
  22. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 117
  23. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
  24. 2020Bayesian Deep Ensembles via the Neural Tangent Kernelcited by 90
  25. 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
  26. 2019Harnessing the Power of Infinitely Wide Deep Nets on Small-data Taskscited by 87
  27. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 87
  28. 2019Non-Gaussian processes and neural networks at finite widthscited by 78
  29. 2018A jamming transition from under- to over-parametrization affects loss landscape and generalizationcited by 65
  30. 2019A Mean Field Theory of Batch Normalizationcited by 58
  31. 2020Exact posterior distributions of wide Bayesian neural networkscited by 29
  32. 2020Towards NNGP-guided Neural Architecture Searchcited by 24
  33. 2021A Neural Tangent Kernel Perspective of GANscited by 21
  34. 2019Disentangling Trainability and Generalization in Deep Neural Networkscited by 16
  35. 2021Meta-Learning with Neural Tangent Kernelscited by 15
  36. 2020Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limitcited by 12
  37. 2022You Only Linearize Once: Tangents Transpose to Gradientscited by 10
  38. 2020Infinitely Wide Graph Convolutional Networks: Semi-supervised Learning via Gaussian Processescited by 8

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 16; 2024: 11; 2025: 18; 2026: 9.

62 papers.

Sort:

By area:

  1. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  2. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 118
  3. 2022Efficient Dataset Distillation Using Random Feature Approximationcited by 111
  4. 2023Structure-free Graph Condensation: From Large-scale Graphs to Condensed Graph-free Datacited by 71
  5. 2023Deep networks for system identification: a Surveycited by 35
  6. 2022A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernelcited by 30
  7. 2024AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Searchcited by 25
  8. 2022Neural Tangent Kernel: A Surveycited by 22
  9. 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
  10. 2024Understanding Linear Probing then Fine-tuning Language Models from NTK Perspectivecited by 21
  11. 2022Fast Neural Kernel Embeddings for General Activationscited by 18
  12. 2022Sampling-based inference for large linear models, with application to linearised Laplacecited by 18
38 more
  1. 2023Variational Linearized Laplace Approximation for Bayesian Deep Learningcited by 17
  2. 2022Few-shot Backdoor Attacks via Neural Tangent Kernelscited by 15
  3. 2023Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillationcited by 14
  4. 2023KECOR: Kernel Coding Rate Maximization for Active 3D Object Detectioncited by 13
  5. 2025Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomizationcited by 13
  6. 2024On the Diminishing Returns of Width for Continual Learningcited by 12
  7. 2023The Interpolating Information Criterion for Overparameterized Modelscited by 11
  8. 2023Stochastic Marginal Likelihood Gradients using Neural Tangent Kernelscited by 10
  9. 2024Finding Lottery Tickets in Vision Models via Data-driven Spectral Foresight Pruningcited by 10
  10. 2023Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrumcited by 9
  11. 2023Metric Flows with Neural Networkscited by 9
  12. 2024What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinementcited by 9
  13. 2024Beimingwu: A Learnware Dock Systemcited by 8
  14. 2024Wilsonian Renormalization of Neural Network Gaussian Processescited by 8
  15. 2024Inductive Gradient Adjustment For Spectral Bias In Implicit Neural Representationscited by 8
  16. 2023Efficient kernel surrogates for neural network-based regressioncited by 7
  17. 2024NTK-Guided Few-Shot Class Incremental Learningcited by 7
  18. 2023Fast-NTK: Parameter-Efficient Unlearning for Large-Scale Modelscited by 5
  19. 2023Towards a Phenomenological Understanding of Neural Networks: Datacited by 4
  20. 2023A PAC-Bayesian Perspective on the Interpolating Information Criterioncited by 4
  21. 2025Rethink the Role of Deep Learning towards Large-scale Quantum Systemscited by 4
  22. 2024Train Faster, Perform Better: Modular Adaptive Training in Over-Parameterized Modelscited by 3
  23. 2025Determinant Estimation under Memory Constraints and Neural Scaling Lawscited by 3
  24. 2025How Learnable Grids Recover Fine Detail in Low Dimensions: A Neural Tangent Kernel Analysis of Multigrid Parametric Encodingscited by 3
  25. 2025Uncertainty-Aware Diagnostics for Physics-Informed Machine Learningcited by 3
  26. 2025Fast Generalized Neural Tangent Kernel Statistics via Trace Estimationcited by 3
  27. 2025Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weightscited by 2
  28. 2025Per-example gradients: a new frontier for understanding and improving optimizerscited by 2
  29. 2025Convergence and Sketching-Based Efficient Computation of Neural Tangent Kernel Weights in Physics-Based Losscited by 2
  30. 2026The Global Empirical NTK: Self-Referential Bias and Dimensionality of Gradient Descent Learningcited by 2
  31. 2024Neural varifolds: an aggregate representation for quantifying the geometry of point cloudscited by 1
  32. 2025Accelerating Natural Gradient Descent for PINNs with Randomized Numerical Linear Algebracited by 1
  33. 2025Feature learning is decoupled from generalization in high capacity neural networkscited by 1
  34. 2025Approximating Full Conformal Prediction for Neural Network Regression with Gauss-Newton Influencecited by 1
  35. 2025From Sublinear to Linear: Local Convergence in Finite-Width Networks via Locally Polyak-Lojasiewicz Regionscited by 1
  36. 2026A Theoretical Framework for LLM Fine-tuning Using Early Stopping for Non-random Initializationcited by 1
  37. 2026Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tailcited by 1
  38. 2023Controlled Descent Trainingcited by 0

50 most cited of 62 All 62 (JSON)