mlsanityData to 7 Oct 2026

On the Neural Tangent Kernel of Equilibrium Models

arXiv 2310.14062 ↗First posted 21 Oct 2023cs.LG cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

This work studies the neural tangent kernel (NTK) of the deep equilibrium (DEQ) model, a practical ``infinite-depth'' architecture which directly computes the infinite-depth limit of a weight-tied network via root-finding. Even though the NTK of a fully-connected neural network can be stochastic if its width and depth both tend to infinity simultaneously, we show that contrarily a DEQ model still enjoys a deterministic NTK despite its width and depth going to infinity at the same time under mild conditions. Moreover, this deterministic NTK can be found efficiently via root-finding.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 23 of its 34 references that are in the map. First posted on arXiv 21 Oct 2023.

Builds on

23 of 34 references.

References · 23

  1. 2018Neural Ordinary Differential Equationscited by 3,733
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2017Deep Neural Networks as Gaussian Processescited by 617
  4. 2019Deep Equilibrium Modelscited by 603
  5. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  6. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  7. 2016On the Expressive Power of Deep Neural Networkscited by 302
  8. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  9. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  10. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  11. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  12. 2020Multiscale Deep Equilibrium Modelscited by 149
11 more
  1. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
  2. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 139
  3. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
  4. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
  5. 2020Monotone operator equilibrium networkscited by 100
  6. 2020The Recurrent Neural Tangent Kernelcited by 67
  7. 2020Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspectivecited by 60
  8. 2018Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networkscited by 44
  9. 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
  10. 2019Dynamical Isometry and a Mean Field Theory of LSTMs and GRUscited by 17
  11. 2019Order and Chaos: NTK views on DNN Normalization, Checkerboard and Boundary Artifactscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 2; 2026: 3.

7 papers.

  1. 2023TorchDEQ: A Library for Deep Equilibrium Modelscited by 21
  2. 2024Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixturescited by 6
  3. 2025Fast Generalized Neural Tangent Kernel Statistics via Trace Estimationcited by 3
  4. 2026The Global Empirical NTK: Self-Referential Bias and Dimensionality of Gradient Descent Learningcited by 2
  5. 2025Expressive Power of Implicit Models: Rich Equilibria and Test-Time Scalingcited by 1
  6. 2026Lipschitz Multiscale Deep Equilibrium Models: A Theoretically Guaranteed and Accelerated Approachcited by 1
  7. 2026Center-Manifold Reduction of Learning at Bifurcations: Interference and Rich Learning in Recurrent Neural Networkscited by 0