mlsanityData to 14 Oct 2026

Kernel Regression with Infinite-Width Neural Networks on Millions of Examples

arXiv 2303.05420 ↗First posted 9 Mar 2023stat.ML cs.CV cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Neural kernels have drastically increased performance on diverse and nonstandard data modalities but require significantly more compute, which previously limited their application to smaller datasets. In this work, we address this by massively parallelizing their computation across many GPUs. We combine this with a distributed, preconditioned conjugate gradients algorithm to enable kernel regression at a large scale (i.e. up to five million examples). Using this approach, we study scaling laws of several neural kernels across many orders of magnitude for the CIFAR-5m dataset. Using data augmentation to expand the original CIFAR-10 training dataset by a factor of 20, we obtain a test accuracy of 91.2% (SotA for a pure kernel method). Moreover, we explore neural kernels on other data modalities, obtaining results on protein and small molecule prediction tasks that are competitive with SotA methods.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 43 of its 60 references that are in the map. First posted on arXiv 9 Mar 2023.

Builds on

43 of 60 references.

References · 43

  1. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  4. 2018How Powerful are Graph Neural Networks?cited by 4,393
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  7. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  8. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  9. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,051
  10. 2017Deep Neural Networks as Gaussian Processescited by 622
  11. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  12. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
31 more
  1. 2013Gaussian Processes for Big Datacited by 383
  2. 2021Explaining Neural Scaling Lawscited by 363
  3. 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 329
  4. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  5. 2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230
  6. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 218
  7. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 210
  8. 2015Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)cited by 177
  9. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 167
  10. 2019Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernelscited by 161
  11. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
  12. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 149
  13. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  14. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 143
  15. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 138
  16. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 121
  17. 2019Exact Gaussian Processes on a Million Data Pointscited by 119
  18. 2020Kernel methods through the roof: handling billions of points efficientlycited by 89
  19. 2017FALKON: An Optimal Large Scale Kernel Methodcited by 84
  20. 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
  21. 2020Neural Kernels Without Tangentscited by 56
  22. 2021Pooling Architecture Search for Graph Classificationcited by 36
  23. 2021Scaling Neural Tangent Kernels via Sketching and Random Featurescited by 32
  24. 2017Diving into the shallows: a computational perspective on large-scale shallow learningcited by 31
  25. 2021Scaling Laws for Deep Learningcited by 26
  26. 2021Reliable Graph Neural Networks for Drug Discovery Under Distributional Shiftcited by 22
  27. 2021Kernel Interpolation for Scalable Online Gaussian Processescited by 18
  28. 2022Fast Neural Kernel Embeddings for General Activationscited by 18
  29. 2020Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limitcited by 12
  30. 2022Low-Precision Arithmetic for Fast Gaussian Processescited by 10
  31. 2022Convolutional Xformers for Visioncited by 5

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 6; 2025: 1.

9 papers.

  1. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 38
  2. 2023Mechanism of feature learning in convolutional neural networkscited by 17
  3. 2024Bayesian Optimization via Continual Variational Last Layer Trainingcited by 13
  4. 2023Convolutional Deep Kernel Machinescited by 3
  5. 2024Learning epidemic trajectories through Kernel Operator Learning: from modelling to optimal controlcited by 1
  6. 2024Flexible Infinite-Width Graph Convolutional Neural Networkscited by 0
  7. 2024Efficient Two-Stage Gaussian Process Regression Via Automatic Kernel Search and Subsamplingcited by 0
  8. 2024Stochastic Kernel Regularisation Improves Generalisation in Deep Kernel Machinescited by 0
  9. 2025Uncertainty Quantification From Scaling Laws in Deep Neural Networkscited by 0