mlsanityData to 7 Oct 2026

The Recurrent Neural Tangent Kernel

arXiv 2006.10246 ↗First posted 18 Jun 2020cs.LG stat.MLcited by 67

Not on the map. Why?

Abstract (arXiv)

The study of deep neural networks (DNNs) in the infinite-width limit, via the so-called neural tangent kernel (NTK) approach, has provided new insights into the dynamics of learning, generalization, and the impact of initialization. One key DNN architecture remains to be kernelized, namely, the recurrent neural network (RNN). In this paper we introduce and study the Recurrent Neural Tangent Kernel (RNTK), which provides new insights into the behavior of overparametrized RNNs. A key property of the RNTK should greatly benefit practitioners is its ability to compare inputs of different length. To this end, we characterize how the RNTK weights different time steps to form its output under different initialization parameters and nonlinearity choices. A synthetic and 56 real-world data experiments demonstrate that the RNTK offers significant performance gains over other kernels, including standard NTKs, across a wide array of data sets.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 16; 2022: 13; 2023: 14; 2024: 9; 2025: 7; 2026: 7.

67 papers.

Sort:

By area:

  1. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  2. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  3. 2023Deep networks for system identification: a Surveycited by 35
  4. 2021A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNscited by 33
  5. 2021Deep Active Learning by Leveraging Training Dynamicscited by 32
  6. 2022Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initializationcited by 30
  7. 2023Neural signature kernels as infinite-width-depth-limits of controlled ResNetscited by 25
  8. 2021A Neural Tangent Kernel Perspective of GANscited by 21
  9. 2023Neural Network Field Theories: Non-Gaussianity, Actions, and Localitycited by 21
  10. 2021Shift Invariance Can Reduce Adversarial Robustnesscited by 16
  11. 2021Neural Tangent Kernel Empowered Federated Learningcited by 15
  12. 2022Few-shot Backdoor Attacks via Neural Tangent Kernelscited by 15
38 more
  1. 2023How connectivity structure shapes rich and lazy learning in neural circuitscited by 15
  2. 2023Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Modelscited by 14
  3. 2023On the Convergence of Encoder-only Shallow Transformerscited by 13
  4. 2021Implicit Bias of Linear RNNscited by 11
  5. 2024Towards Infinite-Long Prefix in Transformercited by 10
  6. 2024Learning Beyond Pattern Matching? Assaying Mathematical Understanding in LLMscited by 9
  7. 2022Kernel Methods and Multi-layer Perceptrons Learn Linear Models in High Dimensionscited by 8
  8. 2022Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Studycited by 7
  9. 2023On the Neural Tangent Kernel of Equilibrium Modelscited by 7
  10. 2024Convergence of Gradient Descent for Recurrent Neural Networks: A Nonasymptotic Analysiscited by 7
  11. 2021On the Provable Generalization of Recurrent Neural Networkscited by 6
  12. 2024Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixturescited by 6
  13. 2023An Exact Kernel Equivalence for Finite Classification Modelscited by 5
  14. 2021A Neural Tangent Kernel Perspective of Infinite Tree Ensemblescited by 4
  15. 2021New Insights into Graph Convolutional Networks using Neural Tangent Kernelscited by 4
  16. 2022Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecturecited by 4
  17. 2024NTK-DFL: Enhancing Decentralized Federated Learning in Heterogeneous Settings via Neural Tangent Kernelcited by 4
  18. 2021Asymptotics of Ridge Regression in Convolutional Modelscited by 3
  19. 2023Representation Learning Dynamics of Self-Supervised Modelscited by 3
  20. 2024"Lossless" Compression of Deep Neural Networks: A High-dimensional Neural Tangent Kernel Approachcited by 3
  21. 2024Unlocking the Theory Behind Scaling 1-Bit Neural Networkscited by 3
  22. 2025Global Convergence and Rich Feature Learning in L-Layer Infinite-Width Neural Networks under μP Parametrizationcited by 3
  23. 2025Fast Generalized Neural Tangent Kernel Statistics via Trace Estimationcited by 3
  24. 2021Ensemble perspective for understanding temporal credit assignmentcited by 2
  25. 2021Experiments with Rich Regime Training for Deep Learningcited by 2
  26. 2022Multi-model Ensemble Analysis with Neural Network Gaussian Processescited by 2
  27. 2022NeuroView-RNN: It's About Timecited by 2
  28. 2022Single-level Adversarial Data Synthesis based on Neural Tangent Kernelscited by 2
  29. 2024Gamma-VAE: Curvature regularized variational autoencoders for uncovering emergent low dimensional geometric structure in high dimensional datacited by 2
  30. 2025Make Haste Slowly: A Theory of Emergent Structured Mixed Selectivity in Feature Learning ReLU Networkscited by 2
  31. 2025Revisiting Glorot Initialization for Long-Range Linear Recurrencescited by 2
  32. 2026The Global Empirical NTK: Self-Referential Bias and Dimensionality of Gradient Descent Learningcited by 2
  33. 2021Weighted Neural Tangent Kernel: A Generalized and Improved Network-Induced Kernelcited by 1
  34. 2022There is a Singularity in the Loss Landscapecited by 1
  35. 2022Embedded Ensembles: Infinite Width Limit and Operating Regimescited by 1
  36. 2023Global Optimality of Elman-type RNN in the Mean-Field Regimecited by 1
  37. 2023An Adaptive Tangent Feature Perspective of Neural Networkscited by 1
  38. 2023Changing the Kernel During Training Leads to Double Descent in Kernel Regressioncited by 1

50 most cited of 67 All 67 (JSON)