mlsanityData to 7 Oct 2026

Neural Kernels Without Tangents

arXiv 2003.02237 ↗First posted 4 Mar 2020cs.LG stat.MLcited by 56

Not on the map. Why?

Abstract (arXiv)

We investigate the connections between neural networks and simple building blocks in kernel space. In particular, using well established feature space tools such as direct sum, averaging, and moment lifting, we present an algebra for creating "compositional" kernels from bags of features. We show that these operations correspond to many of the building blocks of "neural tangent kernels (NTK)". Experimentally, we show that there is a correlation in test error between neural network architectures and the associated kernels. We construct a simple neural network architecture using only 3x3 convolutions, 2x2 average pooling, ReLU, and optimized with SGD and MSE loss that achieves 96% accuracy on CIFAR10, and whose corresponding compositional kernel achieves 90% accuracy. We also use our constructions to investigate the relative performance of neural networks, NTKs, and compositional kernels in the small dataset regime. In particular, we find that compositional kernels outperform NTKs and neural networks outperform both kernel methods.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 21; 2022: 14; 2023: 16; 2024: 4; 2025: 1.

56 papers.

Sort:

  1. 2021Explaining Neural Scaling Lawscited by 353
  2. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 215
  3. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  4. 2022Efficient Dataset Distillation Using Random Feature Approximationcited by 111
  5. 2023Discover and Cure: Concept-aware Mitigation of Spurious Correlationcited by 71
  6. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  7. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  8. 2022Limitations of the NTK for Understanding Generalization in Deep Learningcited by 34
  9. 2021A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNscited by 33
  10. 2021How rotational invariance of common kernels prevents generalization in high dimensionscited by 32
  11. 2022Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functionscited by 32
  12. 2021Approximation and Learning with Deep Convolutional Models: a Kernel Perspectivecited by 31
38 more
  1. 2022Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensionscited by 23
  2. 2021Interpolating Classifiers Make Few Mistakescited by 22
  3. 2022Neural Tangent Kernel: A Surveycited by 22
  4. 2021The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspectivecited by 21
  5. 2021Learning with convolution and pooling operations in kernel methodscited by 21
  6. 2022What Can the Neural Tangent Kernel Tell Us About Adversarial Robustness?cited by 21
  7. 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 21
  8. 2023Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and widthcited by 19
  9. 2023Six Lectures on Linearized Neural Networkscited by 19
  10. 2022Fast Neural Kernel Embeddings for General Activationscited by 18
  11. 2023A Rainbow in Deep Network Black Boxescited by 18
  12. 2021Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networkscited by 17
  13. 2023More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatorycited by 17
  14. 2022Can we achieve robustness from data alone?cited by 16
  15. 2023Mechanism of feature learning in convolutional neural networkscited by 15
  16. 2021Generalization Guarantees for Neural Architecture Search with Train-Validation Splitcited by 14
  17. 2023On the Parameterization of Second-Order Optimization Effective Towards the Infinite Widthcited by 14
  18. 2024Towards Understanding Inductive Bias in Transformers: A View From Infinitycited by 13
  19. 2022Largest Eigenvalues of the Conjugate Kernel of Single-Layered Neural Networkscited by 12
  20. 2023Toward Large Kernel Modelscited by 12
  21. 2023Kernel Regression with Infinite-Width Neural Networks on Millions of Examplescited by 9
  22. 2023Initialization Matters: Privacy-Utility Analysis of Overparameterized Neural Networkscited by 9
  23. 2021The Unreasonable Effectiveness of Patches in Deep Convolutional Kernels Methodscited by 8
  24. 2021Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applicationscited by 8
  25. 2022Predicting Properties of Quantum Systems with Conditional Generative Modelscited by 8
  26. 2021Tilting the playing field: Dynamical loss functions for machine learningcited by 7
  27. 2021Random Features for the Neural Tangent Kernelcited by 6
  28. 2022Minimalistic Unsupervised Learning with the Sparse Manifold Transformcited by 6
  29. 2023Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Spacecited by 5
  30. 2021Correlated Weights in Infinite Limits of Deep Convolutional Neural Networkscited by 3
  31. 2023Convolutional Deep Kernel Machinescited by 3
  32. 2022Single-level Adversarial Data Synthesis based on Neural Tangent Kernelscited by 2
  33. 2023Neural Networks with Sparse Activation Induced by Large Bias: Tighter Analysis with Bias-Generalized NTKcited by 2
  34. 2023On the Nystrom Approximation for Preconditioning in Kernel Machinescited by 2
  35. 2025A Gap Between the Gaussian RKHS and Neural Networks: An Infinite-Center Asymptotic Analysiscited by 2
  36. 2021Weighted Neural Tangent Kernel: A Generalized and Improved Network-Induced Kernelcited by 1
  37. 2021VC dimension of partially quantized neural networks in the overparametrized regimecited by 1
  38. 2022Deep Regression Ensemblescited by 1

50 most cited of 56 All 56 (JSON)