mlsanityData to 14 Oct 2026

A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernel

arXiv 2206.12543 ↗First posted 25 Jun 2022stat.ML cs.LGcited by 30

Not on the map. Why?

Abstract (arXiv)

Empirical neural tangent kernels (eNTKs) can provide a good understanding of a given network's representation: they are often far less expensive to compute and applicable more broadly than infinite width NTKs. For networks with O output units (e.g. an O-class classifier), however, the eNTK on N inputs is of size NO × NO, taking O((NO)2) memory and up to O((NO)3) computation. Most existing applications have therefore used one of a handful of approximations yielding N × N kernel matrices, saving orders of magnitude of computation, but with limited to no justification. We prove that one such approximation, which we call "sum of logits", converges to the true eNTK at initialization for any network with a wide final "readout" layer. Our experiments demonstrate the quality of this approximation for various uses across a range of settings.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Builds on 36 of its 51 references that are in the map. First posted on arXiv 25 Jun 2022.

Builds on

36 of 51 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  4. 2016Wide Residual Networkscited by 3,070
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  6. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  7. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  8. 2017Deep Neural Networks as Gaussian Processescited by 622
  9. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  10. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  11. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  12. 2018Dataset Distillationcited by 230
24 more
  1. 2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230
  2. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 218
  3. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
  4. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  5. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 167
  6. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
  7. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 151
  8. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 143
  9. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 91
  10. 2020Bayesian Deep Ensembles via the Neural Tangent Kernelcited by 90
  11. 2019Harnessing the Power of Infinitely Wide Deep Nets on Small-data Taskscited by 87
  12. 2017FALKON: An Optimal Large Scale Kernel Methodcited by 84
  13. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  14. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  15. 2022A Framework and Benchmark for Deep Batch Active Learning for Regressioncited by 49
  16. 2021Deep Active Learning by Leveraging Training Dynamicscited by 32
  17. 2015Steps Toward Deep Kernel Methods from Infinite Neural Networkscited by 28
  18. 2020Towards NNGP-guided Neural Architecture Searchcited by 24
  19. 2021A Neural Tangent Kernel Perspective of GANscited by 21
  20. 2019The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Studycited by 16
  21. 2021Meta-Learning with Neural Tangent Kernelscited by 15
  22. 2020Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limitcited by 12
  23. 2022Generalization Through The Lens Of Leave-One-Out Errorcited by 10
  24. 2019The Benefits of Over-parameterization at Initialization in Deep ReLU Networkscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 7; 2025: 12; 2026: 4.

30 papers.

Sort:

By area:

  1. 2024Why Do You Grok? A Theoretical Analysis of Grokking Modular Additioncited by 36
  2. 2024Understanding Linear Probing then Fine-tuning Language Models from NTK Perspectivecited by 22
  3. 2023Differentially Private Neural Tangent Kernels for Privacy-Preserving Data Generationcited by 15
  4. 2025Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Modelcited by 13
  5. 2025Models of Heavy-Tailed Mechanistic Universalitycited by 10
  6. 2023Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrumcited by 9
  7. 2024Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyondcited by 9
  8. 2023Faithful and Efficient Explanations for Neural Networks via Neural Tangent Kernel Surrogate Modelscited by 8
  9. 2024Inductive Gradient Adjustment For Spectral Bias In Implicit Neural Representationscited by 8
  10. 2023Federated Learning on Virtual Heterogeneous Data with Local-global Distillationcited by 6
  11. 2024lpNTK: Better Generalisation with Less Data via Sample Interaction During Learningcited by 6
  12. 2025A Unified Theory of Sinusoidal Activation Families for Implicit Neural Representationscited by 4
18 more
  1. 2025Determinant Estimation under Memory Constraints and Neural Scaling Lawscited by 4
  2. 2025Feature Identification via the Empirical NTKcited by 4
  3. 2025Mining Useful General Data for Low-Resource Domain Adaptationcited by 4
  4. 2023NTKCPL: Active Learning on Top of Self-Supervised Model by Estimating True Coveragecited by 3
  5. 2023How Graph Neural Networks Learn: Lessons from Training Dynamicscited by 3
  6. 2024Train Faster, Perform Better: Modular Adaptive Training in Over-Parameterized Modelscited by 3
  7. 2025LEAD: Exploring Logit Space Evolution for Model Selectioncited by 3
  8. 2025Fast Generalized Neural Tangent Kernel Statistics via Trace Estimationcited by 3
  9. 2023AdaFlood: Adaptive Flood Regularizationcited by 2
  10. 2024Predicting the Encoding Error of SIRENscited by 2
  11. 2025Convergence and Sketching-Based Efficient Computation of Neural Tangent Kernel Weights in Physics-Based Losscited by 2
  12. 2026Learning Dynamics in RL Post-Training for Language Modelscited by 2
  13. 2026The Global Empirical NTK: Self-Referential Bias and Dimensionality of Gradient Descent Learningcited by 2
  14. 2025Towards Understanding the Optimization Mechanisms in Deep Learningcited by 0
  15. 2025Is Supervised Learning Really That Different from Unsupervised?cited by 0
  16. 2025Scalable Data Attribution via Forward-Only Test-Time Inferencecited by 0
  17. 2026Center-Manifold Reduction of Learning at Bifurcations: Interference and Rich Learning in Recurrent Neural Networkscited by 0
  18. 2026Certified Approximation for Interpretable Representer Landmarkscited by 0