mlsanityData to 14 Oct 2026

Reverse Engineering the Neural Tangent Kernel

arXiv 2106.03186 ↗First posted 6 Jun 2021cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

The development of methods to guide the design of neural networks is an important open challenge for deep learning theory. As a paradigm for principled neural architecture design, we propose the translation of high-performing kernels, which are better-understood and amenable to first-principles design, into equivalent network architectures, which have superior efficiency, flexibility, and feature learning. To this end, we constructively prove that, with just an appropriate choice of activation function, any positive-semidefinite dot-product kernel can be realized as either the NNGP or neural tangent kernel of a fully-connected neural network with only one hidden layer. We verify our construction numerically and demonstrate its utility as a design tool for finite fully-connected networks in several experiments.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 30 of its 57 references that are in the map. First posted on arXiv 6 Jun 2021.

Builds on

30 of 57 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  5. 2017Deep Neural Networks as Gaussian Processescited by 622
  6. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  7. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  8. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  9. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  10. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  11. 2016On the Expressive Power of Deep Neural Networkscited by 306
  12. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
18 more
  1. 2016Benefits of depth in neural networkscited by 250
  2. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  3. 2016Deep Information Propagationcited by 195
  4. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 174
  5. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 167
  6. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
  7. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  8. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 151
  9. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  10. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 141
  11. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  12. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 121
  13. 2016Why does deep and cheap learning work so well?cited by 100
  14. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 85
  15. 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
  16. 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
  17. 2020Kernel Alignment Risk Estimator: Risk Prediction from Training Datacited by 54
  18. 2017The power of deeper networks for expressing natural functionscited by 35

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 4; 2024: 1; 2025: 4.

12 papers.

  1. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  2. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  3. 2023Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimensioncited by 26
  4. 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
  5. 2022Fast Neural Kernel Embeddings for General Activationscited by 18
  6. 2023Regularization of polynomial networks for image recognitioncited by 13
  7. 2023Biological Sequence Kernels with Guaranteed Flexibilitycited by 7
  8. 2025Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networkscited by 3
  9. 2025When Bias Pretends to Be Truth: How Spurious Correlations Undermine Hallucination Detection in LLMscited by 3
  10. 2025On the Complexity-Faithfulness Trade-off of Gradient-Based Explanationscited by 2
  11. 2025Issues with Neural Tangent Kernel Approach to Neural Networkscited by 1
  12. 2023Reverse Engineering Deep ReLU Networks An Optimization-based Algorithmcited by 0