Data to 7 Oct 2026

Every Model Learned by Gradient Descent Is Approximately a Kernel Machine

arXiv 2012.00152 ↗First posted 30 Nov 2020cs.LG cs.NE stat.MLcited by 53

Not on the map. Why?

Abstract (arXiv)

Deep learning's successes are often attributed to its ability to automatically discover new representations of the data, rather than relying on handcrafted features like other learning methods. We show, however, that deep networks learned by the standard gradient descent algorithm are in fact mathematically approximately equivalent to kernel machines, a learning method that simply memorizes the data and uses it directly for prediction via a similarity function (the kernel). This greatly enhances the interpretability of deep network weights, by elucidating that they are effectively a superposition of the training examples. The network architecture incorporates knowledge of the target function into the kernel. This improved understanding should lead to better learning algorithms.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 12; 2023: 7; 2024: 9; 2025: 6; 2026: 4.

53 papers.

Sort:

By area:

2021PAQ: 65 Million Probably-Asked Questions and What You Can Do With Themcited by 198
2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
2022The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attentioncited by 45
2021Associative Memories via Predictive Codingcited by 42
2021Computing on Functions Using Randomized Vector Representationscited by 41
2023Learning Support and Trivial Prototypes for Interpretable Image Classificationcited by 38
2022On the Generalization Mystery in Deep Learningcited by 25
2021On the Equivalence between Neural Network and Support Vector Machinecited by 17
2021Data-driven geophysical forecasting: Simple, low-cost, and accurate baselines with kernel methodscited by 15
2021Online Limited Memory Neural-Linear Bandits with Likelihood Matchingcited by 14
2021AutoDO: Robust AutoAugment for Biased Data with Label Noise via Scalable Probabilistic Implicit Differentiationcited by 14
2021Robust Uncertainty Bounds in Reproducing Kernel Hilbert Spaces: A Convex Optimization Approachcited by 14

38 more

2021Learning curves for Gaussian process regression with power-law priors and targetscited by 14
2022Evolution of Neural Tangent Kernels under Benign and Adversarial Trainingcited by 14
2024Learning in PINNs: Phase transition, diffusion equilibrium, and generalizationcited by 13
2025Adaptive kernel predictors from feature-learning infinite limits of neural networkscited by 13
2022A Survey of Automated Data Augmentation Algorithms for Deep Learning-based Image Classification Taskscited by 12
2022Learning from Randomly Initialized Neural Network Featurescited by 11
2024Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyondcited by 9
2021CETransformer: Casual Effect Estimation via Transformer Based Representation Learningcited by 8
2023Faithful and Efficient Explanations for Neural Networks via Neural Tangent Kernel Surrogate Modelscited by 8
2022Analysis of Convolutions, Non-linearity and Depth in Graph Neural Networks using Neural Tangent Kernelcited by 7
2022Efficient Augmentation for Imbalanced Deep Learningcited by 6
2022Can neural networks extrapolate? Discussion of a theorem by Pedro Domingoscited by 6
2024Lurking in the shadows: Unveiling Stealthy Backdoor Attacks against Personalized Federated Learningcited by 6
2023Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Spacecited by 5
2023An Exact Kernel Equivalence for Finite Classification Modelscited by 5
2021ACDC: Online Unsupervised Cross-Domain Adaptationcited by 4
2021New Insights into Graph Convolutional Networks using Neural Tangent Kernelscited by 4
2022The Quantum Path Kernel: a Generalized Quantum Neural Tangent Kernel for Deep Quantum Machine Learningcited by 4
2022TorchNTK: A Library for Calculation of Neural Tangent Kernels of PyTorch Modelscited by 3
2024Dynamic Normativity: Necessary and Sufficient Conditions for Value Alignmentcited by 3
2024Scale generalisation properties of extended scale-covariant and scale-invariant Gaussian derivative networks on image datasets with spatial scaling variationscited by 3
2024An evaluation of LLM code generation capabilities through graded exercisescited by 3
2025ExPLAIND: Unifying Model, Data, and Training Attribution to Study Model Behaviorcited by 3
2022On the optimization and generalization of overparameterized implicit neural networkscited by 2
2025Random feature approximation for general spectral methodscited by 2
2025Instruction-Level Weight Shaping: A Framework for Self-Improving AI Agentscited by 2
2022Interpretable (not just posthoc-explainable) medical claims modeling for discharge placement to prevent avoidable all-cause readmissions or deathcited by 1
2023Towards Understanding How Data Augmentation Works with Imbalanced Datacited by 1
2026On Interpolation Formulas Describing Neural Network Generalizationcited by 1
2021Gradient Kernel Regressioncited by 0
2021Self-supervised denoising for massive noisy imagescited by 0
2023A Quantitative Approach to Predicting Representational Learning and Performance in Neural Networkscited by 0
2023Random feature approximation for general spectral methodscited by 0
2024Empirical influence functions to understand the logic of fine-tuningcited by 0
2024Adversarial Vulnerability as a Consequence of On-Manifold Inseparibilitycited by 0
2024EUGens: Efficient, Unified, and General Dense Layerscited by 0
2025Ambiguity in LLMs is a concept missing problemcited by 0
2025Random at First, Fast at Last: NTK-Guided Fourier Pre-Processing for Tabular DLcited by 0

50 most cited of 53 All 53 (JSON)